The darkgs from tyz1030

darkgs's Introduction

DarkGS: Learning Neural Illumination and 3D Gaussians Relighting for Robotic Exploration in the Dark

"Even with these dark eyes, a gift of the dark night, I go to seek the shining light." --Gu Cheng 1956-1993

Novel-view rendering: Simulating a light cone and re-illuminating the environment.

Please check our videos (Bilibili).

Sister Repo for Camera-Light calibration

The sister repo Neural Light Simulator for light-camera calibration is here.

Install

Installation generally follows vanilla Gaussian Splatting installation.

git clone [email protected]:tyz1030/darkgs.git --recursive

git clone https://github.com/tyz1030/darkgs.git --recursive

Conda environment setup

conda env create --file environment.yml
conda activate darkgs

Also need to install lietorch

pip install git+https://github.com/princeton-vl/lietorch.git

Data

Please find our example data on Google Drive and DropBox.

Make your own data

Please put your RAW images subfolder named "raw". To make COLMAP less struggle, I gamma-curved/manually increased the brightness of the raw images for feature extraction and matching. These corrected images are put in "input" subfolder. We only use "raw" images to build DarkGS.

python3 convert.py -s <path to your own dataset>

Light Calibration

If you are using your own light-camera setup, please calibrate your system using neural light simulator. Then put model_parameters.pth in the root directory.

Quick Start

Train

python train.py -s <path to example dataset>

Visualize with SIRB viewer:

./SIBR_remoteGaussian_app

Then you will be able to steer your light cone by pressing "JKLI" on the keyboard.\

Visualize (a checkpoint) after training:

SIBR_gaussianViewer_app is not compatible with this repo. Please try the following:

python3 viz_chkpt.py -s data/lab1/ -m output/<xxxxxx-xxx> --start_checkpoint output/<xxxxxx-xxx>/chkpnt30000.pth

then in another terminal

./SIBR_remoteGaussian_app

Relighting (I'm working on the release)

Meanwhile, there is no one-true-solution to relighting.
I have ADHD symptoms, so please forgive my slow progress.
One quick hack through is when running viz_chkpt.py, uncomment line 129 in scene/lighting.py. And you will also need to brighten, white balance and gamma correct the final render results to make it look good otherwise it is in blue-greenish RAW format.

Cite

This work is picked up by IROS 2024 as oral presentation! Arxiv

@INPROCEEDINGS{zhang2024darkgs,
  author={Tianyi Zhang and Kaining Huang and Weiming Zhi and Matthew Johnson-Roberson},
  booktitle={2024 International Conference on Intelligent Robots and Systems (IROS)}, 
  title={DarkGS: Learning Neural Illumination and 3D Gaussians Relighting for Robotic Exploration in the Dark}, 
  year={2024}}

Acknowledgement

This work is supported by NOAA.
Copyright 2024 Tianyi Zhang, Carnegie Mellon University. All rights reserved.

darkgs's People

Contributors

Stargazers

Watchers

darkgs's Issues

Train problem

When I run the command python train.py -s .\lab1 , the following result was output in terminal normally.
(darkgs) PS D:\darkgs> python train.py -s .\lab1
Optimizing
Output folder: ./output/ddf3c879-f [21/05 15:37:38]
Tensorboard not available: not logging progress [21/05 15:37:38]
Reading camera 114/114 [21/05 15:37:38]
Loading Training Cameras [21/05 15:37:38]
Loading Test Cameras [21/05 15:37:52]
Number of points at initialisation : 5667 [21/05 15:37:52]

However,the output folder only have

BRDF & Albedo

Hi Zhang, thank you for your great work. I have a question about the paper in ArXiv.
May I ask why you multiply brdf and albedo?

Visualization of training results

How to visualize the results after training with the provided dataset and obtaining corresponding results? I used the same viewer as the 3D GS but couldn't see any results. Is there anything worth noting in this step? Looking forward to your reply.
Thank you very much for your work!

tyz1030 / darkgs Goto Github PK