jiatengliu/fov-3dgs

Official Implementation of FoV-3DGS: Real-Time Neural Rendering through Efficiency-Aware Pruning and Foveated Rendering.

FoV-3DGS

Official Implementation of RTGS: Enabling Real-Time Gaussian Splatting on Mobile Devices Using Efficiency-Guided Pruning and Foveated Rendering. [Paper]

(1) Setup

• Clone the repo

git clone https://github.com/horizon-research/FoV-3DGS.git

• Prepare Dataset

  • Download from: Mip360
  • Download the official preprocessed data from 3D Gaussian Splatting (3DGS): Tank&Temple and DeepBlending
  • put the unzipped scene folders in FoV-3DGS/dataset

• Prepare Dense 3DGS for pruning

  • Original 3DGS: download from 3DGS
  • Mini-Splatting-D: We provide our reprodeuced model of m360 bicycle here, for other scenes, you can reproduce using their code.
  • move the dense model to the scene folder and name it "ms_d", structure under the folder should be like:

  |-- cameras.json
  |-- cfg_args
  |-- chkpnt30000.pth
  |-- input.ply
  `-- point_cloud
      `-- iteration_30000
          `-- point_cloud.ply

• Prepare environment:

  • We use docker:

  # pull the docker (this is for x86 machine, for jetson you will need other prbuilt, see https://github.com/dusty-nv/jetson-containers/tree/master and find one that suitable for tour jetpack.)
  docker pull pytorch/pytorch:2.3.0-cuda11.8-cudnn8-devel
  # run docker
  bash ./run_docker.sh
  # go in to docker and install all submodules
  bash update_submodules.sh

  • install some packages

      pip install plyfile opencv-python matplotlib icecream
      apt-get update
      apt-get install libgl1-mesa-glx libglib2.0-0 -y

(2) Run the pruning & FR (Foveated Rendering) Masking pipeline

# we only leave bicycle, uncomment other scenes for batch test
 python3 combined_training_script.py 

The result will be stored in the scene folder.

(3) Measure Objective Metric for PS=1 model.

python3 quality_eval.py 

The result will be in ./full_eval_results/ours-Q

(4) Generate the FoV model & Measure its FPS

bash batch_ours_fps.sh 

Baselines

• SM (Shared Model) FR: Generate + Measure the Layer-Wise Quality & FPS

# we only leave bicycle, uncomment other scenes for batch test
bash batch_gen_naive_FR.sh # generate SMFR
python3 quality_eval_layers_naive.py # measure qulaity in each layer, result will be in ./layers_eval_results/naiveFR
bash batch_naive_fps.sh #measure fps, result will be in ./fps

• MM (Multi-Model) FR : Generate + Measure the Layer-Wise Quality & FPS this one need LightGS for Pruning Multiple Models, we already include it in our repo

bash batch_pnum_analyzer.sh  # analyze pnum of our model in each layer
cd ../LightGaussian
bash ./batch_gen_mmFR.sh # the result will be in ./MMFR/ours-Q
cd ../fov3dgs
python3 quality_eval_layers_mmfr.py # measure qulaity in each layer, result will be in ./layers_eval_results/MMFR
bash ./batch_mmfr_fps.sh #measure fps, result will be in ./fps

Acknowledgements

• Our 3D Gaussian Splatting (3DGS) related code is based on the work from 3DGS.
• Our Human Visual System (HVS) model code is adapted from the Perception library in Odak.