mbortolon97/6dgs

Code of the paper: 6DGS: 6D Pose Estimation from a Single Image and a 3D Gaussian Splatting Model

6DGS

Project page | Paper

This repository contains a PyTorch implementation for the paper: 6DGS: 6D Pose Estimation from a Single Image and a 3D Gaussian Splatting Model.

Installation

Tested on Ubuntu 22.04 + Pytorch 1.10.1

Install environment:

conda env create --file environment.yml
conda activate gaussian_splatting

Quick start

Setup the data structure

For Tanks&Temples we use the dataset format of NSVF:

Tanks&Temples

The Ignatius object inside the Tanks&Temples dataset contain a malformed intrinsics.txt, here you can find the same file correctly formatted, if you replace the original with this should work without issues.

For Mip-NeRF 360°, it is necessary to download the part 1 of the dataset at:

Mip-NeRF 360°

You can place the datasets where is more convenient to you, but you need to change the location inside tools/launch_all_mip_training.sh and tools/launch_all_tanks_and_temple_training.sh.

Training the base 3DGS model

The training script is located in train.py. To train a single 3DGS model:

python train.py -s [dataset location]

We provide two scripts that it is necessary only to edit with the correct paths to the dataset:

sh tools/launch_all_mip_training.sh
sh tools/launch_all_tanks_and_temple_training.sh

Run the pose estimation

The training and testing script for the pose estimation is located in pretrain_eval_attention.py, for training and testing on all the objects from Mip-NeRF 360:

python3 pretrain_eval_attention.py --exp_path ./output/ --out_path results.json --data_type mip360

For the Tanks Temple objects

python3 pretrain_eval_attention.py --exp_path ./output/ --out_path results.json --data_type tankstemple

Citation

If you find our code or paper helps, please consider citing:

@INPROCEEDINGS{Bortolon20246dgs,
  author = {Bortolon, Matteo and Tsesmelis, Theodore and James, Stuart and Poiesi, Fabio and Del Bue, Alessio},
  title = {6DGS: 6D Pose Estimation from a Single Image and a 3D Gaussian Splatting Model},
  booktitle = {ECCV},
  year = {2024}
}

6dgs's Issues

What directory does the "--exp_path" parameter refer to?

Thank you very much for open-sourcing this project, but I have encountered an issue now.
I would like to ask, what directory does the "--exp_path" parameter refer to? Currently, I can train Gaussian Splatting normally. However, when I try to execute the pose estimation according to the python3 pretrain_eval_attention.py --exp_path ../pose-splatting/output/ --out_path results.json --data_type tankstemple line of your markdown, it prompts me with the error FileNotFoundError: [Errno 2] No such file or directory: '../pose-splatting/output/. So, I modified the path of this parameter to the output directory where my Gaussian Splatting training is completed, which includes the point cloud files for 7000 and 30000 steps. After that, when I execute the above code, it can output the 'results' file normally, but the file is empty. Therefore, I would like to ask, what directory does this "--exp_path" parameter refer to?

Training Scripts for Tanks and Temples

First of all, I would like to express my gratitude for your work and efforts in developing this project. I am experiencing some issues while running the training scripts in your project. When I execute the commands:

sh tools/launch_all_tanks_and_temple_training.sh

I encounter issues that seem to be related to the path settings in tanksandtemples.py:

path_base = "/home/mbortolon/data/datasets/onepose/train_data/0000-catterpillar"
os.makedirs(path_base, exist_ok=True)

Additionally, I am facing issues while training the 'Ignatius' scene. Could you provide some guidance on how to resolve these problems?

Thank you for your help!

Best regards

Can this project be used for real-time 6dof spatial positioning?

Hello, thank you very much for open-sourcing such a great project. I would like to ask you, can this project be used for real-time 6dof spatial positioning? For example, if I already have a scene trained with 3D GS and have the corresponding point cloud files, can I continuously input photos of actions in the above scene and then obtain the corresponding 6dof positioning?

Pose estimation testing on custom 3DGS model

Hi,

Many thanks for your valuable contribution!

I was wondering if your framework allows using custom 3DGS model and testing on it. Many thanks in advance!

Best regards,
Isso

Evaluation metrics on MipNeRF360

Thanks for sharing the great work! I have a small question regarding how you compute the evaluation metrics on MipNeRF360 dataset in Tab1.

"Where 1u is equal to the object’s largest dimension. "

Can you point me to the code where you compute this object's largest dimension to normalize the translation error ?

For the other methods, are you computing the numbers yourself on this dataset or the numbers are taken from some other papers ?

The code seems to be incomplete

Thank you so much for open-source this interesting work. I want to run this code. However, the code seems to be incomplete. Can you give the detailed steps? In addition, the python version also seems to have an impact on the code. I am looking forward to hearing from you.
Best wishes.