Giter Site home page Giter Site logo

penguin-jpg / dngaussian Goto Github PK

View Code? Open in Web Editor NEW

This project forked from fictionarry/dngaussian

0.0 0.0 0.0 5.24 MB

[CVPR'24] DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization

Home Page: https://fictionarry.github.io/DNGaussian/

License: Other

Shell 1.92% C++ 0.17% Python 80.17% C 0.47% Cuda 17.27%

dngaussian's Introduction

DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization

This is the official repository for our CVPR 2024 paper DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization.

Paper | Project | Video

image

Installation

Tested on Ubuntu 18.04, CUDA 11.3, PyTorch 1.12.1

conda env create --file environment.yml
conda activate dngaussian

cd submodules
git clone [email protected]:ashawkey/diff-gaussian-rasterization.git --recursive
git clone https://gitlab.inria.fr/bkerbl/simple-knn.git
pip install ./diff-gaussian-rasterization ./simple-knn

If encountering installation problem of the diff-gaussian-rasterization or gridencoder, you may get some help from gaussian-splatting and torch-ngp.

Evaluation

LLFF

  1. Download LLFF from the official download link.

  2. Generate monocular depths by DPT:

    cd dpt
    python get_depth_map_for_llff_dtu.py --root_path $<dataset_path_for_llff> --benchmark LLFF
  3. Start training and testing:

    # for example
    bash scripts/run_llff.sh data/llff/fern output/llff/fern ${gpu_id}

DTU

  1. Download DTU dataset

    • Download the DTU dataset "Rectified (123 GB)" from the official website, and extract it.
    • Download masks (used for evaluation only) from this link.
  2. Organize DTU for few-shot setting

    bash scripts/organize_dtu_dataset.sh $rectified_path
  3. Format

    • Poses: following gaussian-splatting, run convert.py to get the poses and the undistorted images by COLMAP.
    • Render Path: following LLFF to get the poses_bounds.npy from the COLMAP data. (Optional)
  4. Generate monocular depths by DPT:

    cd dpt
    python get_depth_map_for_llff_dtu.py --root_path $<dataset_path_for_dtu> --benchmark DTU
  5. Set the mask path and the expected output model path in copy_mask_dtu.sh for evaluation. (default: "data/dtu/submission_data/idrmasks" and "output/dtu")

  6. Start training and testing:

    # for example
    bash scripts/run_dtu.sh data/dtu/scan8 output/dtu/scan8 ${gpu_id}

Blender

  1. Download the NeRF Synthetic dataset from here.

  2. Generate monocular depths by DPT:

    cd dpt
    python get_depth_map_for_blender.py --root_path $<dataset_path_for_blender>
  3. Start training and testing:

    # for example
    # there are some special settings for different scenes in the Blender dataset, please refer to "run_blender.sh".
    bash scripts/run_blender.sh data/nerf_synthetic/drums output/blender/drums ${gpu_id}

Reproducing Results

Due to the randomness of the densification process and random initialization, the metrics may be unstable in some scenes, especially PSNR.

Checkpoints and Results

You can download our provided checkpoints from here. These results are reproduced with a lower error tolerance bound to keep aligned with this repo, which is different from what we use in the paper. This could lead to higher metrics but worse visualization.

MVS Point Cloud Initialization

If more stable performance is needed, we recommend trying the dense initialization from FSGS.

Here we provide an example script for LLFF that just modifies a few hyperparameters to adapt our method to this initialization:

# Following FSGS to get the "data/llff/$<scene>/3_views/dense/fused.ply" first
bash scripts/run_llff_mvs.sh data/llff/$<scene> output_dense/$<scene> ${gpu_id}

However, there may still be some randomness.

For reference, the best results we get in two random tests are as follows:

PSNR LPIPS SSIM (SK) SSIM (GS)
19.942 0.228 0.682 0.687

where GS refers to the calculation originally provided by 3DGS, and SK denotes calculated by sklearn which is used in most previous NeRF-based methods.

Customized Dataset

Similar to Gaussian Splatting, our method can read standard COLMAP format datasets. Please customize your sampling rule in scenes/dataset_readers.py, and see how to organize a COLMAP-format dataset from raw RGB images referring to our preprocessing of DTU.

Citation

Consider citing as below if you find this repository helpful to your project:

@article{li2024dngaussian,
   title={DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization},
   author={Li, Jiahe and Zhang, Jiawei and Bai, Xiao and Zheng, Jin and Ning, Xin and Zhou, Jun and Gu, Lin},
   journal={arXiv preprint arXiv:2403.06912},
   year={2024}
}

Acknowledgement

This code is developed on gaussian-splatting with simple-knn and a modified diff-gaussian-rasterization. The implementation of neural renderer are based on torch-ngp. Codes about DPT are partial from SparseNeRF. Thanks for these great projects!

dngaussian's People

Contributors

fictionarry avatar penguin-jpg avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.