guanjunwu/gaussiandreamer

GaussianDreamer: Fast Generation from Text to 3D Gaussian Splatting with Point Cloud Priors

GaussianDreamer: Fast Generation from Text to 3D Gaussian Splatting with Point Cloud Priors

Project Page | arxiv Paper

GaussianDreamer: Fast Generation from Text to 3D Gaussian Splatting with Point Cloud Priors

Taoran Yi¹, Jiemin Fang^2‡,Guanjun Wu³, Lingxi Xie²,
Xiaopeng Zhang²,Wenyu Liu¹, Qi Tian² , Xinggang Wang^1‡✉

¹School of EIC, HUST  ²Huawei Inc.   ³School of CS, HUST  

^‡Project lead. ^✉Corresponding author.

In recent times, the generation of 3D assets from text prompts has shown impressive results. Both 2D and 3D diffusion models can generate decent 3D objects based on prompts. 3D diffusion models have good 3D consistency, but their quality and generalization are limited as trainable 3D data is expensive and hard to obtain. 2D diffusion models enjoy strong abilities of generalization and fine generation, but the 3D consistency is hard to guarantee. This paper attempts to bridge the power from the two types of diffusion models via the recent explicit and efficient 3D Gaussian splatting representation. A fast 3D generation framework, named as GaussianDreamer, is proposed, where the 3D diffusion model provides point cloud priors for initialization and the 2D diffusion model enriches the geometry and appearance. Operations of noisy point growing and color perturbation are introduced to enhance the initialized Gaussians. Our GaussianDreamer can generate a high-quality 3D instance within 25 minutes on one GPU, much faster than previous methods, while the generated instances can be directly rendered in real time.

🦾 Updates

• 10/24/2023: Release the results initialized using SMPL. See the Project Page for details.
• 10/21/2023: Fixed some installation issues, thanks to Sikuang Li, Tawfik Boujeh, and ashawkey. You can view the detailed information in branch diff.
• 10/16/2023: The rough code has been released, and there may still be some issues. Please feel free to raise issues.

🏁 Get Started

Installation Install 3D Gaussian Splatting and Shap-E as fellow:

pip install torch==2.0.1+cu117 torchvision==0.15.2+cu117 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu117
pip install ninja
pip install -r requirements.txt

git clone https://github.com/hustvl/GaussianDreamer.git 
cd GaussianDreamer

pip install ./gaussiansplatting/submodules/diff-gaussian-rasterization
pip install ./gaussiansplatting/submodules/simple-knn

git clone https://github.com/openai/shap-e.git
cd shap-e
pip install -e .

Download finetuned Shap-E by Cap3D, and put it in ./load

Quickstart

python launch.py --config configs/gaussiandreamer-sd.yaml --train --gpu 0 system.prompt_processor.prompt="a fox"

📑 Citation

If you find this repository/work helpful in your research, welcome to cite the paper and give a ⭐. Some source code of ours is borrowed from Threestudio, 3DGS, depth-diff-gaussian-rasterization. We sincerely appreciate the excellent works of these authors.

@article{GaussianDreamer,
        title={GaussianDreamer: Fast Generation from Text to 3D Gaussian Splatting with Point Cloud Priors},
        author={Taoran Yi and Jiemin Fang and Guanjun Wu and Lingxi Xie and Xiaopeng Zhang and Wenyu Liu and Qi Tian and Xinggang Wang},
        journal={arxiv:2310.08529},
        year={2023}
        }