muskie82/monogs

[CVPR'24 Highlight & Best Demo Award] Gaussian Splatting SLAM

Gaussian Splatting SLAM

*Hidenobu Matsuki · *Riku Murai · Paul H.J. Kelly · Andrew J. Davison

(* Equal Contribution)

CVPR 2024 (Highlight)

Paper | Video | Project Page

This software implements dense SLAM system presented in our paper Gaussian Splatting SLAM in CVPR'24. The method demonstrates the first monocular SLAM solely based on 3D Gaussian Splatting (left), which also supports Stereo/RGB-D inputs (middle/right).

Note

• In an academic paper, please refer to our work as Gaussian Splatting SLAM or MonoGS for short (this repo's name) to avoid confusion with other works.
• Differential Gaussian Rasteriser with camera pose gradient computation is available here.
• [New] Speed-up version of our code is available in dev.speedup branch, It achieves up to 10fps on monocular fr3/office sequence while keeping consistent performance (tested on RTX4090/i9-12900K). The code will be merged into the main branch after further refactoring and testing.

Getting Started

Installation

git clone https://github.com/muskie82/MonoGS.git --recursive
cd MonoGS

Setup the environment.

conda env create -f environment.yml
conda activate MonoGS

Depending on your setup, please change the dependency version of pytorch/cudatoolkit in environment.yml by following this document.

Our test setup were:

• Ubuntu 20.04: pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.6
• Ubuntu 18.04: pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.3

Quick Demo

bash scripts/download_tum.sh
python slam.py --config configs/mono/tum/fr3_office.yaml

You will see a GUI window pops up.

Downloading Datasets

Running the following scripts will automatically download datasets to the ./datasets folder.

TUM-RGBD dataset

bash scripts/download_tum.sh

Replica dataset

bash scripts/download_replica.sh

EuRoC MAV dataset

bash scripts/download_euroc.sh

Run

Monocular

python slam.py --config configs/mono/tum/fr3_office.yaml

RGB-D

python slam.py --config configs/rgbd/tum/fr3_office.yaml

python slam.py --config configs/rgbd/replica/office0.yaml

Or the single process version as

python slam.py --config configs/rgbd/replica/office0_sp.yaml

Stereo (experimental)

python slam.py --config configs/stereo/euroc/mh02.yaml

Live demo with Realsense

First, you'll need to install pyrealsense2. Inside the conda environment, run:

pip install pyrealsense2

Connect the realsense camera to the PC on a USB-3 port and then run:

python slam.py --config configs/live/realsense.yaml

We tested the method with Intel Realsense d455. We recommend using a similar global shutter camera for robust camera tracking. Please avoid aggressive camera motion, especially before the initial BA is performed. Check out the first 15 seconds of our YouTube video to see how you should move the camera for initialisation. We recommend to use the code in dev.speed-up branch for live demo.

Evaluation

To evaluate our method, please add --eval to the command line argument:

python slam.py --config configs/mono/tum/fr3_office.yaml --eval

This flag will automatically run our system in a headless mode, and log the results including the rendering metrics.

Reproducibility

There might be minor differences between the released version and the results in the paper. Please bear in mind that multi-process performance has some randomness due to GPU utilisation. We run all our experiments on an RTX 4090, and the performance may differ when running with a different GPU.

Acknowledgement

This work incorporates many open-source codes. We extend our gratitude to the authors of the software.

• 3D Gaussian Splatting
• Differential Gaussian Rasterization
• SIBR_viewers
• Tiny Gaussian Splatting Viewer
• Open3D
• Point-SLAM

License

MonoGS is released under a LICENSE.md. For a list of code dependencies which are not property of the authors of MonoGS, please check Dependencies.md.

Citation

If you found this code/work to be useful in your own research, please considering citing the following:

@inproceedings{Matsuki:Murai:etal:CVPR2024,
  title={{G}aussian {S}platting {SLAM}},
  author={Hidenobu Matsuki and Riku Murai and Paul H. J. Kelly and Andrew J. Davison},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2024}
}

monogs's Issues

About realsensed435i

when i run python slam.py --config configs/live/realsense.yaml
Traceback (most recent call last):
File "slam.py", line 252, in
slam = SLAM(config, save_dir=save_dir)
File "slam.py", line 56, in init
model_params, model_params.source_path, config=config
File "/home/xiangchengliu/MonoGS/utils/dataset.py", line 496, in load_dataset
return RealsenseDataset(args, path, config)
File "/home/xiangchengliu/MonoGS/utils/dataset.py", line 434, in init
self.profile = self.pipeline.start(self.config)
RuntimeError: Couldn't resolve requests

The visualization of Gaussians is strange

Thank you for the great work.
When I run the system, the Gaussian plot looks very strange.
Does anyone know the cause of this problem.
THX.

How to run code with GT pose?

It seems like there is no config item in config file to let code run with GT pose (skip the tracking procedure and use gt pose directly). How should we achieve this?

Failed to install diff_gaussian_rasterization on Windows (Solved)

system environment

• Windows 10
• Visual Studio 2022
• cuda==11.8.0
• pytorch==2.0.0

Error message

I get the following error message when I run pip install. \submodules\diff-gaussian-rasterization-w-pose\

D:\GaussianSLAM\MonoGS\submodules\diff-gaussian-rasterization-w-pose\cuda_rasterizer\backward.cu(554): error: expected an expression
                detected during instantiation of "void renderCUDA<C>(const uint2 *, const uint32_t *, int, int, const float *, const float2 *, 
const float4 *, const float *, const float *, const float *, const uint32_t *, const float *, const float *, float3 *, float4 *, float *, float *, float *) 
[with C=3U]"
      (882): here

      1 error detected in the compilation of "D:/GaussianSLAM/MonoGS/submodules/diff-gaussian-rasterization-w-pose/cuda_rasterizer/backward.cu".

      backward.cu

In line 554 of /cuda_rasterizer/backward.cu

548    template <typename group_t, typename... Lists>
549    __device__ void render_cuda_reduce_sum(group_t g, Lists... lists) {
550      int lane = g.thread_rank();
551      g.sync();
552    
553      for (int i = g.size() / 2; i > 0; i /= 2) {
554        (...,
555         reduce_helper(                                      //  ERROR HERE
556             lane, i, lists)); // Fold expression: apply reduce_helper for each list
557        g.sync();
558      }
559   }

Solution

Modify the render_cuda_reduce_sum function as follows

template <typename group_t, typename... Lists>
__device__ void render_cuda_reduce_sum(group_t g, Lists... lists) {
  int lane = g.thread_rank();
  g.sync();

  for (int i = g.size() / 2; i > 0; i /= 2) {
//     (...,reduce_helper(lane, i, lists)); // Fold expression: apply reduce_helper for each list
    (void)std::initializer_list<int>{(reduce_helper(lane, i, lists), 0)...};
    g.sync();
  }
}

Camera Tracking result on Replica for Monocular SLAM Results

Hello, the paper shows the results of the RGB-D SLAM on the Replica datasets.
However, I obtained Monocular SLAM tracking results by adjusting the configuration file.
For example, on room0 it's about 4cm, I don't know where I went wrong and the result is unexpected.

Therefore, can you provide the complete evaluation results or guide us on how to adjust configuration file to get correct results.

Thx so much

how can i run it with a RGB-D camera?

Hello, first of all, thank you for open-sourcing such excellent work. I have a seemingly silly question: How can I use an RGB-D camera and your open-source code to achieve the effects shown in your demo videos?

ImportError: Cannot load backend 'TkAgg' which requires the 'tk' interactive framework, as 'headless' is currently running

A error occurs when “ python slam.py --config configs/mono/tum/fr3_office.yaml”

extend to object 6dof tracking

Hi,

Awesome work.
I wonder how feasible it is to extend it to handle object pose estimation using this.
Sounds like we can do an inverted slam for the object if we have masked segmented objects in multiple images?

Build diff-gaussian-rasterization failed.

• OS: Windows 10
• CUDA outside: CUDA 11.8
• CUDA inside: CUDA 11.6

Processing f:\test\monogs\submodules\diff-gaussian-rasterization
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: diff_gaussian_rasterization
  Building wheel for diff_gaussian_rasterization (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [62 lines of output]
      running bdist_wheel
      D:\Programs\miniconda3\envs\monogs\lib\site-packages\torch\utils\cpp_extension.py:411: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
        warnings.warn(msg.format('we could not find ninja.'))
      running build
      running build_py
      copying diff_gaussian_rasterization\__init__.py -> build\lib.win-amd64-cpython-37\diff_gaussian_rasterization
      running build_ext
      D:\Programs\miniconda3\envs\monogs\lib\site-packages\torch\utils\cpp_extension.py:813: UserWarning: The detected CUDA version (11.8) has a minor version mismatch with the version that was used to compile PyTorch (11.6). Most likely this shouldn't be a problem.
        warnings.warn(CUDA_MISMATCH_WARN.format(cuda_str_version, torch.version.cuda))
      building 'diff_gaussian_rasterization._C' extension
      "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\bin\nvcc" -c cuda_rasterizer/backward.cu -o build\temp.win-amd64-cpython-37\Release\cuda_rasterizer/backward.obj -ID:\Programs\miniconda3\envs\monogs\lib\site-packages\torch\include -ID:\Programs\miniconda3\envs\monogs\lib\site-packages\torch\include\torch\csrc\api\include -ID:\Programs\miniconda3\envs\monogs\lib\site-packages\torch\include\TH -ID:\Programs\miniconda3\envs\monogs\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\include" -ID:\Programs\miniconda3\envs\monogs\include -ID:\Programs\miniconda3\envs\monogs\Include "-ID:\Programs\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\ATLMFC\include" "-ID:\Programs\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.2\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt" -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -IF:\Test\MonoGS\submodules\diff-gaussian-rasterization\third_party/glm/ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --use-local-env
      F:\Test\MonoGS\submodules\diff-gaussian-rasterization\cuda_rasterizer\auxiliary.h(151): warning #177-D: variable "p_proj" was declared 
but never referenced
     
      C:/Program Files (x86)/Windows Kits/10/include/10.0.19041.0/um\winnt.h(22866): warning #174-D: expression has no effect
     
      C:/Program Files (x86)/Windows Kits/10/include/10.0.19041.0/um\winuser.h(14630): warning #108-D: signed bit field of length 1
     
      C:/Program Files (x86)/Windows Kits/10/include/10.0.19041.0/um\winuser.h(14631): warning #108-D: signed bit field of length 1
     
      C:/Program Files (x86)/Windows Kits/10/include/10.0.19041.0/um\wincrypt.h(21791): warning #1835-D: attribute "dllimport" does not apply here
     
      C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared\rpcndr.h(719): warning #108-D: signed bit field of length 1
     
      C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared\rpcndr.h(720): warning #108-D: signed bit field of length 1
     
      C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared\rpcndr.h(721): warning #108-D: signed bit field of length 1
     
      C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared\rpcndr.h(722): warning #108-D: signed bit field of length 1
     
      C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared\rpcndr.h(723): warning #108-D: signed bit field of length 1
     
      C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared\rpcndr.h(724): warning #108-D: signed bit field of length 1
     
      C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared\rpcndr.h(725): warning #108-D: signed bit field of length 1
     
      C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared\rpcndr.h(726): warning #108-D: signed bit field of length 1
     
      C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared\rpcndr.h(727): warning #108-D: signed bit field of length 1
     
      C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared\rpcndr.h(728): warning #108-D: signed bit field of length 1
     
      C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared\rpcndr.h(729): warning #108-D: signed bit field of length 1
     
      C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared\rpcndr.h(730): warning #108-D: signed bit field of length 1
     
      C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared\rpcndr.h(731): warning #108-D: signed bit field of length 1
     
      F:\Test\MonoGS\submodules\diff-gaussian-rasterization\cuda_rasterizer\backward.cu(471): warning #177-D: variable "c" was declared but never referenced
                detected during instantiation of "void preprocessCUDA<C>(int, int, int, const float3 *, const int *, const float *, const __nv_bool *, const glm::vec3 *, const glm::vec4 *, float, const float *, const float *, const float *, const glm::vec3 *, const float3 *, glm::vec3 *, float *, float *, float *, float *, glm::vec3 *, glm::vec4 *, float *) [with C=3]"
      (838): here
     
      F:\Test\MonoGS\submodules\diff-gaussian-rasterization\cuda_rasterizer\backward.cu(472): warning #177-D: variable "d" was declared but never referenced
                detected during instantiation of "void preprocessCUDA<C>(int, int, int, const float3 *, const int *, const float *, const __nv_bool *, const glm::vec3 *, const glm::vec4 *, float, const float *, const float *, const float *, const glm::vec3 *, const float3 *, glm::vec3 *, float *, float *, float *, float *, glm::vec3 *, glm::vec4 *, float *) [with C=3]"
      (838): here
     
      F:\Test\MonoGS\submodules\diff-gaussian-rasterization\cuda_rasterizer\backward.cu(554): error: expected an expression
                detected during instantiation of "void renderCUDA<C>(const uint2 *, const uint32_t *, int, int, const float *, const float2 *, const float4 *, const float *, const float *, const float *, const uint32_t *, const float *, const float *, float3 *, float4 *, float *, float *, float *) [with C=3U]"
      (882): here
     
      1 error detected in the compilation of "cuda_rasterizer/backward.cu".
      backward.cu
      error: command 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.8\\bin\\nvcc.exe' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for diff_gaussian_rasterization
  Running setup.py clean for diff_gaussian_rasterization
Failed to build diff_gaussian_rasterization
Installing collected packages: diff_gaussian_rasterization
  Running setup.py install for diff_gaussian_rasterization ... error
  error: subprocess-exited-with-error

How to visialize pointcloud using GUI after training

Hello! Thanks for your great work.

I wonder how to use GUI to visialize using the GUI you design.
I have the result-----point_cloud.ply file of the fr3_desk scene.

Trank you very much.

Question about ground truth pose use with 3DGS

Dear authors,

Thank you for your amazing work !

I would like to know how you managed to use the estimated camera poses with 3DGS.
I am trying to benchmark the ICL NUIM dataset with the base 3DGS implementation using the ground truth poses.
I wrote a custom data loader to replace the COLMAP estimated R, T and intrinsics with the ground truth ones for each camera objects. I also took note of the C2W to W2C transformation between the COLMAP norm and my dataset.
So to recap, I'm training the 3DGS with GT poses and intrinsics, GT colored point-cloud for initialization.

However I can't seem to get any result in doing so (see the screenshot attached). My camera poses seems off even though I am using the GT poses.

There is something I am clearly missing but can't seem to figure out what it is to get my camera poses right.
Have you faced similar problems and/or do you know how to solve this issue ?

Thanks a lot,

Hugo

Gaussian Pruning for Monocular case

It's a very good job and the code is very elegant

Q1: In the paper "Guassian Insertion and Pruning" get the second paragraph

If the Gaussians inserted within the last 3 keyframes are unobserved by at least 3 other frames, we prune them out as they are geometrically unstable.

In the map update of backend, the part code:

if prune:
  if len(current_window) == self.config["Training"]["window_size"]:
      prune_mode = self.config["Training"]["prune_mode"]
      prune_coviz = 3
      self.gaussians.n_obs.fill_(0)
      for window_idx, visibility in self.occ_aware_visibility.items():
          self.gaussians.n_obs += visibility.cpu()
      to_prune = None
      if prune_mode == "odometry":
          to_prune = self.gaussians.n_obs < 3
          # make sure we don't split the gaussians, break here.
      if prune_mode == "slam":
          # only prune keyframes which are relatively new
          sorted_window = sorted(current_window, reverse=True)
          mask = self.gaussians.unique_kfIDs >= sorted_window[2]
          if not self.initialized:
              mask = self.gaussians.unique_kfIDs >= 0
          to_prune = torch.logical_and(
              self.gaussians.n_obs <= prune_coviz, mask
          )
      if to_prune is not None and self.monocular:
          self.gaussians.prune_points(to_prune.cuda())
          for idx in range((len(current_window))):
              current_idx = current_window[idx]
              self.occ_aware_visibility[current_idx] = (
                  self.occ_aware_visibility[current_idx][~to_prune]
              )
      if not self.initialized:
          self.initialized = True
          Log("Initialized SLAM")
      # # make sure we don't split the gaussians, break here.
  return False

According to my understanding, the sort_window[2] means last 3 keyframes in the paper, the self.gaussians.unique_kfIDs means all gaussians unique id in the map. So, how should I understand by at least 3 other frames in the paper. Isn’t it the remaining frames in the current window except for the three newly inserted key frames?

Q2: It's also related to self.gaussians.unique_kfIDs

According to the every time unique_kfIDs updates in extend_from_pcd() :

new_unique_kfIDs = torch.ones((new_xyz.shape[0])).int() * kf_id

It is bound to the ID of the current keyframe. It also appears in other functions: densify_and_clone, densify_and_split, densification_postfix, prune_points
Therefore, the unique here is mainly for the geometric judgment of the newly inserted Gaussian points in the last three frames in the current window for monocular case.

I just started reading the code and I don't know if that's the right way to understand it.

Why there is no f_rest property in the ply file?

I’ve been working with the generated PLY files and have observed a few issues that I’d like to bring to your attention.

Firstly, I noticed that the generated PLY files lack the f_rest property. This is unlike the original Gaussian splatting PLY files which do include this property. Could you please explain why this discrepancy exists?

Secondly, I’ve found that the quality of the rendered images is not up to par. The images appear to be of poor quality. Could you provide some insight into why this might be happening?

Is there any loop closing or Bundle adjustment ?

Thank you for your great work!

Since I'm a undergraduated student, my questions may be clumsy. I appreciate it if you can take this into consideration.

The paper said "it didn't use loop closing" and even in evaluation, the models that used loop closing performed much better, so why didn't you use it?

Is the diffrerent purpose of the model ?

Can we plot the GT trajectory in the GUI?

Hey guys congrats on the great work.

I think it would be quite helpful to plot the GT trajectory as well in the GUI to allow for comparison. I can take a look at doing it myself, but I'm still familiarizing myself with the code, so if you could give me your thoughts on what my options are for implementing this at a high level that would be great.

if you already have the code for doing that, please point me to it, thank you!

Questions about code writing

Hello, sorry to bother you, I am looking at the code recently and I have two questions
Q1: Regarding the process of running the code, I found that if I use a monocular photo without depth (that is, RGB), the depth parameter is not used in the slam process. So if I run a Depth estimation estimates the depth without monocular photos (RGB). If this is similar to RGBD for slam, will the effect be better?
Q2: I want to improve my ability to write code, so I want to write code based on your code. I would like to ask if there is a good direction for my reference!
Thank you for your hard work!

Question relates to Point Clound Generation

This is such nice work!!!!
I am a newbie in GS splatting. I have several questions.

1. As I read in the original paper of GS, they derived the Point Cloud (x,y,z) from SfM and stored it in COLMAP format. In MonoGS, How is Point Cloud generated (I think it is from photometric difference right ? ). Where is the part of the code?
2. I read the code and guessed that the SLAM front-end and back-end run in parallel right? Can I have a short pseudo code from inputs to outputs for better understanding of codes?

Many thanks.

Issue on pip install submodules/simple-knn

When I try to install the package simple-knn, it shows blow.

Processing ./submodules/simple-knn
Preparing metadata (setup.py) ... done
Building wheels for collected packages: simple-knn
Building wheel for simple-knn (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [29 lines of output]
running bdist_wheel
/well/rittscher/users/ycr745/.conda/envs/MonoGS/lib/python3.11/site-packages/torch/utils/cpp_extension.py:500: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
warnings.warn(msg.format('we could not find ninja.'))
running build
running build_ext
/well/rittscher/users/ycr745/.conda/envs/MonoGS/lib/python3.11/site-packages/torch/utils/cpp_extension.py:388: UserWarning:

                                 !! WARNING !!
  
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  Your compiler (g++ 4.8.5) may be ABI-incompatible with PyTorch!
  Please use a compiler that is ABI-compatible with GCC 5.0 and above.
  See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.
  
  See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
  for instructions on how to install GCC 5 or higher.
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  
                                !! WARNING !!
  
    warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
  /well/rittscher/users/ycr745/.conda/envs/MonoGS/lib/python3.11/site-packages/torch/utils/cpp_extension.py:425: UserWarning: There are no g++ version bounds defined for CUDA version 12.1
    warnings.warn(f'There are no {compiler_name} version bounds defined for CUDA version {cuda_str_version}')
  building 'simple_knn._C' extension
  creating build
  creating build/temp.linux-x86_64-cpython-311
  gcc -pthread -B /well/rittscher/users/ycr745/.conda/envs/MonoGS/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /well/rittscher/users/ycr745/.conda/envs/MonoGS/include -fPIC -O2 -isystem /well/rittscher/users/ycr745/.conda/envs/MonoGS/include -fPIC -I/well/rittscher/users/ycr745/.conda/envs/MonoGS/lib/python3.11/site-packages/torch/include -I/well/rittscher/users/ycr745/.conda/envs/MonoGS/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/well/rittscher/users/ycr745/.conda/envs/MonoGS/lib/python3.11/site-packages/torch/include/TH -I/well/rittscher/users/ycr745/.conda/envs/MonoGS/lib/python3.11/site-packages/torch/include/THC -I/usr/local/cuda/include -I/well/rittscher/users/ycr745/.conda/envs/MonoGS/include/python3.11 -c ext.cpp -o build/temp.linux-x86_64-cpython-311/ext.o -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++17
  gcc: error: unrecognized command line option '-std=c++17'
  error: command '/usr/bin/gcc' failed with exit code 1
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for simple-knn
Running setup.py clean for simple-knn
Failed to build simple-knn
ERROR: Could not build wheels for simple-knn, which is required to install pyproject.toml-based projects

Does anyone know what's the problem?

There is one error in GUI with multiprocess

The hardware environment is a single 3090. And timing of this error is uncertain. Although the slam process continues later, the stuck GUI cause the memory overflow. This may be the reason why Issue#7 cannot run.

Test on custom data

Hi thanks again for the amazing work,

Is there a way to test the SLAM system on custom datasets ?
Can you indicate me the procedure to do so ?

Thanks in advance,
Best regards,

Hugo

GaussianRasterizationSettings.__new__() got an unexpected keyword argument 'projmatrix_raw'

Hi thanks for your great work on Gaussian splatting SLAM.

I am encountering an issue while trying to test your implementation :
MonoGS: saving results in result/datasets_tum/2024-03-06-13-49-41 FEngine (64 bits) created at 0x56021834d430 (threading is enabled) FEngine resolved backend: OpenGL MonoGS: Resetting the system Process Process-3: Traceback (most recent call last): File "/home/hleblond/anaconda3/envs/MonoGS/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/home/hleblond/anaconda3/envs/MonoGS/lib/python3.11/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/home/hleblond/Bureau/PhD/MonoGS/utils/slam_backend.py", line 407, in run self.initialize_map(cur_frame_idx, viewpoint) File "/home/hleblond/Bureau/PhD/MonoGS/utils/slam_backend.py", line 89, in initialize_map render_pkg = render( ^^^^^^^ File "/home/hleblond/Bureau/PhD/MonoGS/gaussian_splatting/gaussian_renderer/__init__.py", line 58, in render raster_settings = GaussianRasterizationSettings( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: GaussianRasterizationSettings.__new__() got an unexpected keyword argument 'projmatrix_raw'

Did someone already encounter this issue ?

Thanks in advance,
Best regards,

Hugo

Upload to a server through my PC

Due to limited computing power of my PC, I am unable to perform this on-board training.
I attempt to connect the Realsense camera to my PC and upload in real-time to a server I leased.
Both my PC and server have public IPs. Could you please advise on how I can achieve this?

CUBLAS_STATUS_NOT_SUPPORTED

Running

python slam.py --config configs/mono/tum/fr3_office.yaml

results in the error

Traceback (most recent call last):
  File "/home/fractal/anaconda3/envs/MonoGS/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/fractal/anaconda3/envs/MonoGS/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/fractal/workspaces/MonoGS/utils/slam_backend.py", line 407, in run
    self.initialize_map(cur_frame_idx, viewpoint)
  File "/home/fractal/workspaces/MonoGS/utils/slam_backend.py", line 90, in initialize_map
    viewpoint, self.gaussians, self.pipeline_params, self.background
  File "/home/fractal/workspaces/MonoGS/gaussian_splatting/gaussian_renderer/__init__.py", line 66, in render
    projmatrix=viewpoint_camera.full_proj_transform,
  File "/home/fractal/workspaces/MonoGS/utils/camera_utils.py", line 102, in full_proj_transform
    self.projection_matrix.unsqueeze(0)
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling `cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, num_batches)`

The conda environment is configured exactly as prescribed in your README.

How the input video is initialized as 3D gaussians?

I am sorry to bother you again.
As the paper mentions, the original 3DGS algorithm initializes the system by using Colmap to changes the input video/images into sparse point-cloud and then initialize the point-cloud into 3D gaussians.
the picture is from 《3D Gaussian Splatting for Real-Time Radiance Field Rendering》
But when I read the code, I didn't find the process of converting the input video(RGB/RGB-D) to sparse point-cloud.
Without sparse point-cloud from Colmap, How can you initialize the input video(RGB/RGB-D) into 3D gaussians?
So, I am confused about how the input is initialized as 3D gaussians?
Thanks!

How to save the 3DGS result

Hi, thanks for your work! I ran the code, but not found the result of 3DGS, How to save and show it again?

result blow:
.
├── config.yml
├── plot
├── point_cloud
└── psnr

How to run in a single 3090?

It seems that the config of replica can not run in a single 3090 due to cuda out of memory,which parameter can be changed to run in a single 3090 with 24GiB memory total?

How to View Result After SLAM

Thanks for your outstanding work!
I have a remote computer with GPU 3090 but without a graphical interface. I got results on that (a point_cloud.ply), and I'm wondering if there is a way that I can view the results locally on my own computer with GPU, using your GUI code.
But I didn't find any readme about that, can you provide a simple guide?

failed to open the demo becuase of opengl compile error

Hi!
I tested the first demo

python slam.py --config configs/mono/tum/fr3_office.yaml

And I got the error above and nothing displayed.

MonoGS: saving results in results/datasets_tum/2024-03-15-14-39-15
FEngine (64 bits) created at 0x3770bc90 (threading is enabled)
FEngine resolved backend: OpenGL
Process Process-4:
Traceback (most recent call last):
  File "/opt/miniconda3/envs/MonoGS/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/opt/miniconda3/envs/MonoGS/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/workspace/gui/slam_gui.py", line 687, in run
    win = SLAM_GUI(params_gui)
  File "/workspace/gui/slam_gui.py", line 66, in __init__
    self.g_renderer = OpenGLRenderer(self.g_camera.w, self.g_camera.h)
  File "/workspace/gui/gl_render/render_ogl.py", line 71, in __init__
    os.path.join(cur_path, "shaders/gau_frag.glsl"),
  File "/workspace/gui/gl_render/util.py", line 90, in load_shaders
    shaders.compileShader(vertex_shader, GL_VERTEX_SHADER),
  File "/opt/miniconda3/envs/MonoGS/lib/python3.7/site-packages/OpenGL/GL/shaders.py", line 241, in compileShader
    shaderType,
OpenGL.GL.shaders.ShaderCompilationError: ("Shader compile failure (0): b'0:1(10): error: GLSL 4.30 is not supported. Supported versions are: 1.10, 1.20, 1.30, 1.40, 1.00 ES, and 3.00 ES\\n'", [b'#version 430 core\n\n#define SH_C0 0.28209479177387814f\n#define SH_C1 0.4886025119029199f\n\n#define SH_C2_0 1.0925484305920792f\n#define SH_C2_1 -1.0925484305920792f\n#define SH_C2_2 0.31539156525252005f\n#define SH_C2_3 -1.0925484305920792f\n#define SH_C2_4 0.5462742152960396f\n\n#define SH_C3_0 -0.5900435899266435f\n#define SH_C3_1 2.890611442640554f\n#define SH_C3_2 -0.4570457994644658f\n#define SH_C3_3 0.3731763325901154f\n#define SH_C3_4 -0.4570457994644658f\n#define SH_C3_5 1.445305721320277f\n#define SH_C3_6 -0.5900435899266435f\n\nlayout(location = 0) in vec2 position;\n// layout(location = 1) in vec3 g_pos;\n// layout(location = 2) in vec4 g_rot;\n// layout(location = 3) in vec3 g_scale;\n// layout(location = 4) in vec3 g_dc_color;\n// layout(location = 5) in float g_opacity;\n\n\n#define POS_IDX 0\n#define ROT_IDX 3\n#define SCALE_IDX 7\n#define OPACITY_IDX 10\n#define SH_IDX 11\n\nlayout (std430, binding=0) buffer gaussian_data {\n\tfloat g_data[];\n\t// compact version of following data\n\t// vec3 g_pos[];\n\t// vec4 g_rot[];\n\t// vec3 g_scale[];\n\t// float g_opacity[];\n\t// vec3 g_sh[];\n};\nlayout (std430, binding=1) buffer gaussian_order {\n\tint gi[];\n};\n\nuniform mat4 view_matrix;\nuniform mat4 projection_matrix;\nuniform vec3 hfovxy_focal;\nuniform vec3 cam_pos;\nuniform int sh_dim;\nuniform float scale_modifier;\nuniform int render_mod;  // > 0 render 0-ith SH dim, -1 depth, -2 bill board, -3 gaussian\n\nout vec3 color;\nout float alpha;\nout vec3 conic;\nout vec2 coordxy;  // local coordinate in quad, unit in pixel\n\nmat3 computeCov3D(vec3 scale, vec4 q)  // should be correct\n{\n    mat3 S = mat3(0.f);\n    S[0][0] = scale.x;\n\tS[1][1] = scale.y;\n\tS[2][2] = scale.z;\n\tfloat r = q.x;\n\tfloat x = q.y;\n\tfloat y = q.z;\n\tfloat z = q.w;\n\n    mat3 R = mat3(\n\t\t1.f - 2.f * (y * y + z * z), 2.f * (x * y - r * z), 2.f * (x * z + r * y),\n\t\t2.f * (x * y + r * z), 1.f - 2.f * (x * x + z * z), 2.f * (y * z - r * x),\n\t\t2.f * (x * z - r * y), 2.f * (y * z + r * x), 1.f - 2.f * (x * x + y * y)\n\t);\n\n    mat3 M = S * R;\n    mat3 Sigma = transpose(M) * M;\n    return Sigma;\n}\n\nvec3 computeCov2D(vec4 mean_view, float focal_x, float focal_y, float tan_fovx, float tan_fovy, mat3 cov3D, mat4 viewmatrix)\n{\n    vec4 t = mean_view;\n    // why need this? Try remove this later\n    float limx = 1.3f * tan_fovx;\n    float limy = 1.3f * tan_fovy;\n    float txtz = t.x / t.z;\n    float tytz = t.y / t.z;\n    t.x = min(limx, max(-limx, txtz)) * t.z;\n    t.y = min(limy, max(-limy, tytz)) * t.z;\n\n    mat3 J = mat3(\n        focal_x / t.z, 0.0f, -(focal_x * t.x) / (t.z * t.z),\n\t\t0.0f, focal_y / t.z, -(focal_y * t.y) / (t.z * t.z),\n\t\t0, 0, 0\n    );\n    mat3 W = transpose(mat3(viewmatrix));\n    mat3 T = W * J;\n\n    mat3 cov = transpose(T) * transpose(cov3D) * T;\n    // Apply low-pass filter: every Gaussian should be at least\n\t// one pixel wide/high. Discard 3rd row and column.\n\tcov[0][0] += 0.3f;\n\tcov[1][1] += 0.3f;\n    return vec3(cov[0][0], cov[0][1], cov[1][1]);\n}\n\nvec3 get_vec3(int offset)\n{\n\treturn vec3(g_data[offset], g_data[offset + 1], g_data[offset + 2]);\n}\nvec4 get_vec4(int offset)\n{\n\treturn vec4(g_data[offset], g_data[offset + 1], g_data[offset + 2], g_data[offset + 3]);\n}\n\nvoid main()\n{\n\tint boxid = gi[gl_InstanceID];\n\tint total_dim = 3 + 4 + 3 + 1 + sh_dim;\n\tint start = boxid * total_dim;\n\tvec4 g_pos = vec4(get_vec3(start + POS_IDX), 1.f);\n    vec4 g_pos_view = view_matrix * g_pos;\n    vec4 g_pos_screen = projection_matrix * g_pos_view;\n\tg_pos_screen.xyz = g_pos_screen.xyz / g_pos_screen.w;\n    g_pos_screen.w = 1.f;\n\t// early culling\n\tif (any(greaterThan(abs(g_pos_screen.xyz), vec3(1.3))))\n\t{\n\t\tgl_Position = vec4(-100, -100, -100, 1);\n\t\treturn;\n\t}\n\tvec4 g_rot = get_vec4(start + ROT_IDX);\n\tvec3 g_scale = get_vec3(start + SCALE_IDX);\n\tfloat g_opacity = g_data[start + OPACITY_IDX];\n\n    mat3 cov3d = computeCov3D(g_scale * scale_modifier, g_rot);\n    vec2 wh = 2 * hfovxy_focal.xy * hfovxy_focal.z;\n    vec3 cov2d = computeCov2D(g_pos_view, \n                              hfovxy_focal.z, \n                              hfovxy_focal.z, \n                              hfovxy_focal.x, \n                              hfovxy_focal.y, \n                              cov3d, \n                              view_matrix);\n\n    // Invert covariance (EWA algorithm)\n\tfloat det = (cov2d.x * cov2d.z - cov2d.y * cov2d.y);\n\tif (det == 0.0f)\n\t\tgl_Position = vec4(0.f, 0.f, 0.f, 0.f);\n    \n    float det_inv = 1.f / det;\n\tconic = vec3(cov2d.z * det_inv, -cov2d.y * det_inv, cov2d.x * det_inv);\n    \n    vec2 quadwh_scr = vec2(3.f * sqrt(cov2d.x), 3.f * sqrt(cov2d.z));  // screen space half quad height and width\n    vec2 quadwh_ndc = quadwh_scr / wh * 2;  // in ndc space\n    g_pos_screen.xy = g_pos_screen.xy + position * quadwh_ndc;\n    coordxy = position * quadwh_scr;\n    gl_Position = g_pos_screen;\n    \n    alpha = g_opacity;\n\n\tif (render_mod == -1)\n\t{\n\t\tfloat depth = -g_pos_view.z;\n\t\tdepth = depth < 0.05 ? 1 : depth;\n\t\tdepth = 1 / depth;\n\t\tcolor = vec3(depth, depth, depth);\n\t\treturn;\n\t}\n\n\t// Covert SH to color\n\tint sh_start = start + SH_IDX;\n\tvec3 dir = g_pos.xyz - cam_pos;\n    dir = normalize(dir);\n\tcolor = SH_C0 * get_vec3(sh_start);\n\t\n\tif (sh_dim > 3 && render_mod >= 1)  // 1 * 3\n\t{\n\t\tfloat x = dir.x;\n\t\tfloat y = dir.y;\n\t\tfloat z = dir.z;\n\t\tcolor = color - SH_C1 * y * get_vec3(sh_start + 1 * 3) + SH_C1 * z * get_vec3(sh_start + 2 * 3) - SH_C1 * x * get_vec3(sh_start + 3 * 3);\n\n\t\tif (sh_dim > 12 && render_mod >= 2)  // (1 + 3) * 3\n\t\t{\n\t\t\tfloat xx = x * x, yy = y * y, zz = z * z;\n\t\t\tfloat xy = x * y, yz = y * z, xz = x * z;\n\t\t\tcolor = color +\n\t\t\t\tSH_C2_0 * xy * get_vec3(sh_start + 4 * 3) +\n\t\t\t\tSH_C2_1 * yz * get_vec3(sh_start + 5 * 3) +\n\t\t\t\tSH_C2_2 * (2.0f * zz - xx - yy) * get_vec3(sh_start + 6 * 3) +\n\t\t\t\tSH_C2_3 * xz * get_vec3(sh_start + 7 * 3) +\n\t\t\t\tSH_C2_4 * (xx - yy) * get_vec3(sh_start + 8 * 3);\n\n\t\t\tif (sh_dim > 27 && render_mod >= 3)  // (1 + 3 + 5) * 3\n\t\t\t{\n\t\t\t\tcolor = color +\n\t\t\t\t\tSH_C3_0 * y * (3.0f * xx - yy) * get_vec3(sh_start + 9 * 3) +\n\t\t\t\t\tSH_C3_1 * xy * z * get_vec3(sh_start + 10 * 3) +\n\t\t\t\t\tSH_C3_2 * y * (4.0f * zz - xx - yy) * get_vec3(sh_start + 11 * 3) +\n\t\t\t\t\tSH_C3_3 * z * (2.0f * zz - 3.0f * xx - 3.0f * yy) * get_vec3(sh_start + 12 * 3) +\n\t\t\t\t\tSH_C3_4 * x * (4.0f * zz - xx - yy) * get_vec3(sh_start + 13 * 3) +\n\t\t\t\t\tSH_C3_5 * z * (xx - yy) * get_vec3(sh_start + 14 * 3) +\n\t\t\t\t\tSH_C3_6 * x * (xx - 3.0f * yy) * get_vec3(sh_start + 15 * 3);\n\t\t\t}\n\t\t}\n\t}\n\tcolor += 0.5f;\n}\n'], GL_VERTEX_SHADER)

Thanks

RuntimeError: tabulate: failed to synchronize: cudaErrorInvalidConfiguration: invalid configuration argument

Hi, I am testing on an RTX 4090 + i5-13500 setup.
python3 slam.py --config configs/mono/tum/fr3_office.yaml
I'm changing the pose in tracking with the pose calculated using LoFTR. Specifically, the logic of the code is as follows:

def tracking(self, cur_frame_idx, viewpoint):
				prev = self.cameras[cur_frame_idx - self.use_every_n_frames]
        viewpoint.update_RT(prev.R, prev.T)

        frame_curr = viewpoint.original_image.clone()
        frame_prev = load_image(Path(self.dataset.color_paths[cur_frame_idx - self.use_every_n_frames])).to(device=self.device)

				# prealign image using LoFTR
        prealign_matrix = self.prealign_images(frame_prev, frame_curr, cur_frame_idx, extractor=self.extractor, matcher=self.matcher, vis=False)
        prealign_matrix = torch.tensor(prealign_matrix, dtype=torch.float32, device=self.device)
        
        opt_params = []
        ...
        pose_optimizer = torch.optim.Adam(opt_params)

        for tracking_itr in range(self.tracking_itr_num):
            render_pkg = render(
                viewpoint, self.gaussians, self.pipeline_params, self.background
            )
            image, depth, opacity = ...

            pose_optimizer.zero_grad()
            loss_tracking = get_loss_tracking(... )
            loss_tracking.backward()

            with torch.no_grad():
                if tracking_itr == 0:
                    viewpoint.R = prealign_matrix[0:3, 0:3]
                    viewpoint.T = prealign_matrix[0:3, 3]
                    converged = False
                else:
                    pose_optimizer.step()
                    converged = update_pose(viewpoint)

						...
        return render_pkg

During the run, the following error occurred:

MonoGS: Running MonoGS without GUI
MonoGS: Following config will be overriden
MonoGS:         save_results=True
MonoGS:         use_gui=False
MonoGS:         eval_rendering=True
MonoGS: saving results in results/datasets_tum/2024-04-02-15-30-28
MonoGS: Resetting the system
MonoGS: Initialized map
cur_frame_idx    1
cur_frame_idx    2
cur_frame_idx    3
cur_frame_idx    4
cur_frame_idx    5
cur_frame_idx    6
cur_frame_idx    7
cur_frame_idx    8
cur_frame_idx    9
cur_frame_idx    10
cur_frame_idx    11
cur_frame_idx    12
cur_frame_idx    13
cur_frame_idx    14
cur_frame_idx    15
MonoGS: Keyframes lacks sufficient overlap to initialize the map, resetting.
MonoGS: Resetting the system
MonoGS: Initialized map
cur_frame_idx    16
cur_frame_idx    17
cur_frame_idx    18
cur_frame_idx    19
cur_frame_idx    20
cur_frame_idx    21
cur_frame_idx    22
cur_frame_idx    23
cur_frame_idx    24
cur_frame_idx    25
Process Process-3:
Traceback (most recent call last):
  File "slam.py", line 278, in <module>
Traceback (most recent call last):
    slam = SLAM(config, save_dir=save_dir)
  File "slam.py", line 110, in __init__
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/media/user/NERF_4T_02/ws_3dgs/3dgs_slam/MonoGS/utils/slam_backend.py", line 417, in run
    self.add_next_kf(cur_frame_idx, viewpoint, depth_map=depth_map)
  File "/media/user/NERF_4T_02/ws_3dgs/3dgs_slam/MonoGS/utils/slam_backend.py", line 68, in add_next_kf
    self.gaussians.extend_from_pcd_seq(
  File "/media/user/NERF_4T_02/ws_3dgs/3dgs_slam/MonoGS/gaussian_splatting/scene/gaussian_model.py", line 239, in extend_from_pcd_seq
    self.create_pcd_from_image(cam_info, init, scale=scale, depthmap=depthmap)
  File "/media/user/NERF_4T_02/ws_3dgs/3dgs_slam/MonoGS/gaussian_splatting/scene/gaussian_model.py", line 131, in create_pcd_from_image
    return self.create_pcd_from_image_and_depth(cam, rgb, depth, init)
  File "/media/user/NERF_4T_02/ws_3dgs/3dgs_slam/MonoGS/gaussian_splatting/scene/gaussian_model.py", line 185, in create_pcd_from_image_and_depth
    distCUDA2(torch.from_numpy(np.asarray(pcd.points)).float().cuda()),
RuntimeError: tabulate: failed to synchronize: cudaErrorInvalidConfiguration: invalid configuration argument
    self.frontend.run()
  File "/media/user/NERF_4T_02/ws_3dgs/3dgs_slam/MonoGS/utils/slam_frontend.py", line 725, in run
    data = self.frontend_queue.get()
  File "/usr/lib/python3.8/multiprocessing/queues.py", line 116, in get
    return _ForkingPickler.loads(res)
  File "/media/user/NERF_4T_02/ws_3dgs/3dgs_slam/venv/monogs_20240329/lib/python3.8/site-packages/torch/multiprocessing/reductions.py", line 297, in rebuild_storage_fd
    fd = df.detach()
  File "/usr/lib/python3.8/multiprocessing/resource_sharer.py", line 57, in detach
    with _resource_sharer.get_connection(self._id) as conn:
  File "/usr/lib/python3.8/multiprocessing/resource_sharer.py", line 87, in get_connection
    c = Client(address, authkey=process.current_process().authkey)
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 502, in Client
    c = SocketClient(address)
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
FileNotFoundError: [Errno 2] No such file or directory
[W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
[W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]

What could be the reason for this?
Thank you

Installation Error with simple_knn and diff_gaussian_rasterization

Hello,

Thanks for the great work. When I try to create the project env using Win 11 and Cuda 11.6 (added to the path, I checked it from the terminal and I updated the pytroch etc. version accordingly), I encounter an error while trying to install the packages named simple_knn and diff_gaussian_rasterization. This error seems to occur during compilation of packages. While compiling the simple_knn package, I get an error saying "too few arguments for template template parameter" when using the nvcc tool. Similarly, I get an error when using nvcc when compiling the diff_gaussian_rasterization package.

If you want, I can share the detailed output of the errors. Since it is a very long error text, I did not want to throw it away directly. What could be the cause of this problem?

Thanks.

ValueError:Unable to determine Axes to steal space for Colorbar. Either provide the *cax* argument to use as the Axes for the Colorbar, provide the *ax* argument to steal space from it, or add *mappable* to an Axes.

Thank you very much for your excellent work.When I run the TUM dataset,An issue always arises.
How can I resolve this issue?Looking forward to your response, thank you.

How to decrease the required CUDA Memory in MonoGS

Hello,

Thank you so much for releasing this fantastic work! My computer has one RTX 4080 GPU with 16GB memory, when trying to run the system on realsense mode, it has the following error,

RuntimeError: CUDA out of memory. Tried to allocate 35.27 GiB (GPU 0; 15.69 GiB total capacity; 47.88 MiB already allocated; 10.16 GiB free; 60.00 MiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
So I was wondering which parameter I can adjust to make the required CUDA memory no more than my GPU volume in .yaml configuration file, Thank you!

Optimization parameters about exposure_a and exposure_b

While optimizing the camera pose, I noticed that you also optimized exposure_a and exposure_b, which mainly appear in the following places:

image_ab = (torch.exp(viewpoint.exposure_a)) * image + viewpoint.exposure_b

Sorry for my lack of knowledge, I have never seen this done before.
Q1: Why did you consider doing this? Can you give me some reference documents?
Q2: Regarding whether the results have been greatly improved, have you done ablation experiments on the Replilca and TMU datasets?

Thx

Why delta pose has gradients?

Hello
Thanks for your excellent job. While delving into the tracking module, I have a question regarding the pose update process. I noticed that the parameters to be updated include the delta pose and exposure parameters. I understand that exposure parameters are involved in calculating the RGB loss. However, I couldn't find where the delta pose is utilized during rendering or loss computation.
After performing backward and step operations, the pose is updated with gradient disabled. Consequently, should there be no gradient concerning the delta pose?

I am really appreciate if you could provide some guidanceon how the delta pose is optimized within the tracking.

Best regards

When we backward, the loss computation is independent of the delta pose (cam_rot_delta and cam_trans_delta)
https://github.com/muskie82/MonoGS/blob/main/utils/slam_frontend.py#L162

pose_optimizer = torch.optim.Adam(opt_params)
for tracking_itr in range(self.tracking_itr_num):
    render_pkg = render(
        viewpoint, self.gaussians, self.pipeline_params, self.background
    )
    image, depth, opacity = (
        render_pkg["render"],
        render_pkg["depth"],
        render_pkg["opacity"],
    )
    pose_optimizer.zero_grad()
    loss_tracking = get_loss_tracking(
        self.config, image, depth, opacity, viewpoint
    )
    # note
    loss_tracking.backward()

    with torch.no_grad():
        pose_optimizer.step()
        converged = update_pose(viewpoint)

theta and rho are left unused in the rendering.
https://github.com/rmurai0610/diff-gaussian-rasterization-w-pose/blob/main/diff_gaussian_rasterization/__init__.py#L48C1-L87C10

class _RasterizeGaussians(torch.autograd.Function):
    @staticmethod
    def forward(
        ctx,
        means3D,
        means2D,
        sh,
        colors_precomp,
        opacities,
        scales,
        rotations,
        cov3Ds_precomp,
        theta,
        rho,
        raster_settings,
    ):

        # Restructure arguments the way that the C++ lib expects them
        args = (
            raster_settings.bg,
            means3D,
            colors_precomp,
            opacities,
            scales,
            rotations,
            raster_settings.scale_modifier,
            cov3Ds_precomp,
            raster_settings.viewmatrix,
            raster_settings.projmatrix,
            raster_settings.projmatrix_raw,
            raster_settings.tanfovx,
            raster_settings.tanfovy,
            raster_settings.image_height,
            raster_settings.image_width,
            sh,
            raster_settings.sh_degree,
            raster_settings.campos,
            raster_settings.prefiltered,
            raster_settings.debug,
        )

An error occurred when i run the code

When I run the command: python slam.py --config configs/mono/tum/fr3_office.yaml
An error occurred:
MonoGS: saving results in results/datasets_tum/2024-04-01-19-33-17
WARNING: Using soft CircularBuffer (6144 KiB)
FEngine (64 bits) created at 0x55a3be464590 (threading is enabled)
FEngine resolved backend: OpenGL
MonoGS: Resetting the system
MonoGS: Initialized map
MonoGS: Resetting the opacity of non-visible Gaussians
MonoGS: Performing initial BA for initialization
MonoGS: Initialized SLAM
MonoGS: Evaluating ATE at frame: 113
Traceback (most recent call last):
File "/home/marco/MonoGS/slam.py", line 252, in
slam = SLAM(config, save_dir=save_dir)
File "/home/marco/MonoGS/slam.py", line 110, in init
self.frontend.run()
File "/home/marco/MonoGS/utils/slam_frontend.py", line 468, in run
eval_ate(
File "/home/marco/MonoGS/utils/eval_utils.py", line 105, in eval_ate
ate = evaluate_evo(
File "/home/marco/MonoGS/utils/eval_utils.py", line 29, in evaluate_evo
traj_est_aligned = trajectory.align_trajectory(
AttributeError: module 'evo.core.trajectory' has no attribute 'align_trajectory'
MonoGS: Resetting the opacity of non-visible Gaussians
MonoGS: Resetting the opacity of non-visible Gaussians

What is the problem?

How the camera pose is estimated?

Thanks for your great work !
When I read the papre, I find that there is a process of camera pose estimation in SLAM system overview. So I'm trying to know how the camera pose is estimated. However , I just find how the camera pose is optimised but not estimated when reading the papre.

So I'm confused that how the camera pose is estimated ?
This may be a stupid question, but I would like to get an explanation from you.
Thanks !

CUDA error: the launch timed out and was terminated

Thank you for this program! First,I appologize for my poor English. Sorry!
When I successfully build the environment, then I run it. But I meet these problem. At first, It can run some photo,Then it stuck.

(MonoGS) PS D:\MonoGS> python .\slam.py --config .\configs\rgbd\tum\fr1_desk.yaml MonoGS: saving results in results\datasets_tum\2024-04-01-13-12-54 FEngine (64 bits) created at 000002063B5DB050 (threading is enabled) FEngine resolved backend: OpenGL MonoGS: Resetting the system MonoGS: Initialized map Process Process-3: Traceback (most recent call last): File "D:\Conda\envs\MonoGS\lib\multiprocessing\process.py", line 297, in _bootstrap self.run() File "D:\Conda\envs\MonoGS\lib\multiprocessing\process.py", line 99, in run self._target(*self._args, **self._kwargs) File "D:\MonoGS\utils\slam_backend.py", line 382, in run self.map(self.current_window, prune=True, iters=10) File "D:\MonoGS\utils\slam_backend.py", line 193, in map self.config, image, depth, viewpoint, opacity File "D:\MonoGS\utils\slam_utils.py", line 98, in get_loss_mapping return get_loss_mapping_rgbd(config, image_ab, depth, viewpoint) File "D:\MonoGS\utils\slam_utils.py", line 120, in get_loss_mapping_rgbd dtype=torch.float32, device=image.device RuntimeError: CUDA error: an illegal memory access was encountered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Traceback (most recent call last): File ".\slam.py", line 252, in <module> slam = SLAM(config, save_dir=save_dir) File ".\slam.py", line 110, in __init__ self.frontend.run() File "D:\MonoGS\utils\slam_frontend.py", line 392, in run render_pkg = self.tracking(cur_frame_idx, viewpoint) File "D:\MonoGS\utils\slam_frontend.py", line 174, in tracking self.config, image, depth, opacity, viewpoint File "D:\MonoGS\utils\slam_utils.py", line 60, in get_loss_tracking return get_loss_tracking_rgbd(config, image_ab, depth, opacity, viewpoint) File "D:\MonoGS\utils\slam_utils.py", line 80, in get_loss_tracking_rgbd dtype=torch.float32, device=image.device RuntimeError: CUDA error: the launch timed out and was terminated CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

BrokenPipeError: [Errno 32] Broken pipe

Although this error does not terminate the program, I'm concerned with this issue.

Platform: Ubuntu 22.04.4 LTS
GPU: RTX 3090 (GPU memory: 24GiB)
Python version: 3.11.8

My command: python slam.py --config configs/mono/tum/fr1_desk.yaml --eval

MonoGS: Running MonoGS in Evaluation Mode
MonoGS: Following config will be overriden
MonoGS:         save_results=True
MonoGS:         use_gui=False
MonoGS:         eval_rendering=True
MonoGS:         use_wandb=True
MonoGS: saving results in results/tum_rgbd_dataset_freiburg1_desk/2024-03-27-16-45-34
wandb: Currently logged in as: xxx (xxxteam). Use `wandb login --relogin` to force relogin
wandb: wandb version 0.16.5 is available!  To upgrade, please run:
wandb:  $ pip install wandb --upgrade
wandb: Tracking run with wandb version 0.16.4
wandb: Run data is saved locally in /media/omnisky/sdc/2021/zry/GS/gs-slam/wandb/run-20240327_164536-5b9yasoj
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run configs/mono/tum/fr1_desk_2024-03-27-16-45-34
wandb: ⭐️ View project at https://wandb.ai/xxxteam/MonoGS
wandb: 🚀 View run at https://wandb.ai/xxxteam/MonoGS/runs/5b9yasoj
MonoGS: Resetting the system
MonoGS: Initialized map
MonoGS: Resetting the opacity of non-visible Gaussians
MonoGS: Keyframes lacks sufficient overlap to initialize the map, resetting.
MonoGS: Resetting the system
MonoGS: Initialized map
MonoGS: Resetting the opacity of non-visible Gaussians
MonoGS: Performing initial BA for initialization
MonoGS: Initialized SLAM
MonoGS: Evaluating ATE at frame:  80
Eval: RMSE ATE [m] 0.0075907673407301125
Traceback (most recent call last):
  File "/media/omnisky/sdc/2021/zry/GS/gs-slam/slam.py", line 252, in <module>
    slam = SLAM(config, save_dir=save_dir)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/omnisky/sdc/2021/zry/GS/gs-slam/slam.py", line 110, in __init__
    self.frontend.run()
  File "/media/omnisky/sdc/2021/zry/GS/gs-slam/utils/slam_frontend.py", line 468, in run
    eval_ate(
  File "/media/omnisky/sdc/2021/zry/GS/gs-slam/utils/eval_utils.py", line 105, in eval_ate
    ate = evaluate_evo(
          ^^^^^^^^^^^^^
  File "/media/omnisky/sdc/2021/zry/GS/gs-slam/utils/eval_utils.py", line 54, in evaluate_evo
    evo.tools.plot.traj_colormap(
  File "/media/omnisky/sdc/2021/zry/anaconda3/envs/zrygs/lib/python3.11/site-packages/evo/tools/plot.py", line 395, in traj_colormap
    cbar = fig.colorbar(
           ^^^^^^^^^^^^^
  File "/media/omnisky/sdc/2021/zry/anaconda3/envs/zrygs/lib/python3.11/site-packages/matplotlib/figure.py", line 1285, in colorbar
    raise ValueError(
ValueError: Unable to determine Axes to steal space for Colorbar. Either provide the *cax* argument to use as the Axes for the Colorbar, provide the *ax* argument to steal space from it, or add *mappable* to an Axes.
wandb: 🚀 View run configs/mono/tum/fr1_desk_2024-03-27-16-45-34 at: https://wandb.ai/xxxteam/MonoGS/runs/5b9yasoj
wandb: ️⚡ View job at https://wandb.ai/xxxteam/MonoGS/jobs/QXJ0aWZhY3RDb2xsZWN0aW9uOjE1MzAxMjMyMw==/version_details/v0
wandb: Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./wandb/run-20240327_164536-5b9yasoj/logs
Exception in thread IntMsgThr:
Traceback (most recent call last):
Exception in thread NetStatThr:
  File "/media/omnisky/sdc/2021/zry/anaconda3/envs/zrygs/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
Traceback (most recent call last):
  File "/media/omnisky/sdc/2021/zry/anaconda3/envs/zrygs/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/media/omnisky/sdc/2021/zry/anaconda3/envs/zrygs/lib/python3.11/threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
    self.run()
  File "/media/omnisky/sdc/2021/zry/anaconda3/envs/zrygs/lib/python3.11/site-packages/wandb/sdk/wandb_run.py", line 300, in check_internal_messages
  File "/media/omnisky/sdc/2021/zry/anaconda3/envs/zrygs/lib/python3.11/threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "/media/omnisky/sdc/2021/zry/anaconda3/envs/zrygs/lib/python3.11/site-packages/wandb/sdk/wandb_run.py", line 268, in check_network_status
    self._loop_check_status(
  File "/media/omnisky/sdc/2021/zry/anaconda3/envs/zrygs/lib/python3.11/site-packages/wandb/sdk/wandb_run.py", line 224, in _loop_check_status
    self._loop_check_status(
  File "/media/omnisky/sdc/2021/zry/anaconda3/envs/zrygs/lib/python3.11/site-packages/wandb/sdk/wandb_run.py", line 224, in _loop_check_status
    local_handle = request()
    local_handle = request()
                                      ^^^^^^^^^^^^^^^
^^  File "/media/omnisky/sdc/2021/zry/anaconda3/envs/zrygs/lib/python3.11/site-packages/wandb/sdk/interface/interface.py", line 795, in deliver_network_status
^
  File "/media/omnisky/sdc/2021/zry/anaconda3/envs/zrygs/lib/python3.11/site-packages/wandb/sdk/interface/interface.py", line 803, in deliver_internal_messages
    return self._deliver_network_status(status)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/omnisky/sdc/2021/zry/anaconda3/envs/zrygs/lib/python3.11/site-packages/wandb/sdk/interface/interface_shared.py", line 500, in _deliver_network_status
    return self._deliver_internal_messages(internal_message)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/omnisky/sdc/2021/zry/anaconda3/envs/zrygs/lib/python3.11/site-packages/wandb/sdk/interface/interface_shared.py", line 506, in _deliver_internal_messages
    return self._deliver_record(record)
    return self._deliver_record(record)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/omnisky/sdc/2021/zry/anaconda3/envs/zrygs/lib/python3.11/site-packages/wandb/sdk/interface/interface_shared.py", line 449, in _deliver_record
    handle = mailbox._deliver_record(record, interface=self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/omnisky/sdc/2021/zry/anaconda3/envs/zrygs/lib/python3.11/site-packages/wandb/sdk/interface/interface_shared.py", line 449, in _deliver_record
  File "/media/omnisky/sdc/2021/zry/anaconda3/envs/zrygs/lib/python3.11/site-packages/wandb/sdk/lib/mailbox.py", line 455, in _deliver_record
    handle = mailbox._deliver_record(record, interface=self)
    interface._publish(record)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/omnisky/sdc/2021/zry/anaconda3/envs/zrygs/lib/python3.11/site-packages/wandb/sdk/lib/mailbox.py", line 455, in _deliver_record
  File "/media/omnisky/sdc/2021/zry/anaconda3/envs/zrygs/lib/python3.11/site-packages/wandb/sdk/interface/interface_sock.py", line 51, in _publish
    interface._publish(record)
  File "/media/omnisky/sdc/2021/zry/anaconda3/envs/zrygs/lib/python3.11/site-packages/wandb/sdk/interface/interface_sock.py", line 51, in _publish
    self._sock_client.send_record_publish(record)
    self._sock_client.send_record_publish(record)
  File "/media/omnisky/sdc/2021/zry/anaconda3/envs/zrygs/lib/python3.11/site-packages/wandb/sdk/lib/sock_client.py", line 221, in send_record_publish
  File "/media/omnisky/sdc/2021/zry/anaconda3/envs/zrygs/lib/python3.11/site-packages/wandb/sdk/lib/sock_client.py", line 221, in send_record_publish
    self.send_server_request(server_req)
  File "/media/omnisky/sdc/2021/zry/anaconda3/envs/zrygs/lib/python3.11/site-packages/wandb/sdk/lib/sock_client.py", line 155, in send_server_request
    self.send_server_request(server_req)
  File "/media/omnisky/sdc/2021/zry/anaconda3/envs/zrygs/lib/python3.11/site-packages/wandb/sdk/lib/sock_client.py", line 155, in send_server_request
    self._send_message(msg)
  File "/media/omnisky/sdc/2021/zry/anaconda3/envs/zrygs/lib/python3.11/site-packages/wandb/sdk/lib/sock_client.py", line 152, in _send_message
    self._send_message(msg)
  File "/media/omnisky/sdc/2021/zry/anaconda3/envs/zrygs/lib/python3.11/site-packages/wandb/sdk/lib/sock_client.py", line 152, in _send_message
    self._sendall_with_error_handle(header + data)
  File "/media/omnisky/sdc/2021/zry/anaconda3/envs/zrygs/lib/python3.11/site-packages/wandb/sdk/lib/sock_client.py", line 130, in _sendall_with_error_handle
    self._sendall_with_error_handle(header + data)
  File "/media/omnisky/sdc/2021/zry/anaconda3/envs/zrygs/lib/python3.11/site-packages/wandb/sdk/lib/sock_client.py", line 130, in _sendall_with_error_handle
    sent = self._sock.send(data)
           ^^^^^^^^^^^^^^^^^^^^^
BrokenPipeError: [Errno 32] Broken pipe
    sent = self._sock.send(data)
           ^^^^^^^^^^^^^^^^^^^^^
BrokenPipeError: [Errno 32] Broken pipe
MonoGS: Resetting the opacity of non-visible Gaussians
MonoGS: Resetting the opacity of non-visible Gaussians
MonoGS: Resetting the opacity of non-visible Gaussians

It has not stopped execution until I lost connection to the server after running for 2 hours, while in the paper it is mentioned that rendering is real-time.

Thank you very much.

How the camera pose is initialized and optimized

Thanks for this great work ! ! !
When I read the code , I'm trying to figure out how the camera pose is initialized and optimized . The only part of the code I found that was related to initialization was this one, and it used the ground truth value of the dataset for initialization.
viewpoint = Camera.init_from_dataset( self.dataset, cur_frame_idx, projection_matrix ) and
viewpoint.update_RT(viewpoint.R_gt, viewpoint.T_gt)
So I'm confused as to why the ground truth value is used to initialize, but the camera gradient is still calculated and optimized afterwards? Isn't the initialized truth value the system optimal value?
This may be a stupid question, but I would like to get an explanation from you .Thanks ! !

Environment configuration issue

An error message during the installation process indicates that pip tried to install the submodules/simple-knn directory as a Python package, but because there was neither a setup.py file nor a pyproject.toml file in this directory, the installation failed.
I checked under ./simple-knn, there is no file

ValueError: Unable to determine Axes to steal space for Colorbar.

MonoGS: Evaluating ATE at frame: 106
Eval: RMSE ATE [m] 0.007080079125603161
Traceback (most recent call last):
File "/home/zzq/workspace/MonoGS/slam.py", line 252, in
slam = SLAM(config, save_dir=save_dir)
File "/home/zzq/workspace/MonoGS/slam.py", line 110, in init
self.frontend.run()
File "/home/zzq/workspace/MonoGS/utils/slam_frontend.py", line 468, in run
eval_ate(
File "/home/zzq/workspace/MonoGS/utils/eval_utils.py", line 105, in eval_ate
ate = evaluate_evo(
File "/home/zzq/workspace/MonoGS/utils/eval_utils.py", line 54, in evaluate_evo
evo.tools.plot.traj_colormap(
File "/home/zzq/.local/lib/python3.10/site-packages/evo/tools/plot.py", line 395, in traj_colormap
cbar = fig.colorbar(
File "/home/zzq/.local/lib/python3.10/site-packages/matplotlib/figure.py", line 1285, in colorbar
raise ValueError(
ValueError: Unable to determine Axes to steal space for Colorbar. Either provide the cax argument to use as the Axes for the Colorbar, provide the ax argument to steal space from it, or add mappable to an Axes.
MonoGS: Resetting the opacity of non-visible Gaussians
MonoGS: Resetting the opacity of non-visible Gaussians

I don't know why this situation occurred. I have installed and run the code according to the requirements. After running, only a few cameras can be seen on the graphical interface, and then this situation will occur. That is to say, an error occurred after running for a while.

ImportError: /lib64/libm.so.6: version `GLIBC_2.29' not found

Hi,

When I try to run the demo using

python slam.py --config configs/mono/tum/fr3_office.yaml
, the program outputs the following error

Traceback (most recent call last):
  File "slam.py", line 15, in <module>
    from gui import gui_utils, slam_gui
  File "/home/yingywan/Documents/MonoGS/gui/slam_gui.py", line 19, in <module>
    from gui.gl_render import util, util_gau
  File "/home/yingywan/Documents/MonoGS/gui/gl_render/__init__.py", line 1, in <module>
    from . import render_ogl, util, util_gau
  File "/home/yingywan/Documents/MonoGS/gui/gl_render/render_ogl.py", line 7, in <module>
    from . import util, util_gau
  File "/home/yingywan/Documents/MonoGS/gui/gl_render/util.py", line 1, in <module>
    import glm

ImportError: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /home/xxx/miniconda3/envs/MonoGS/lib/python3.7/site-packages/glm.cpython-37m-x86_64-linux-gnu.so)

I think it is caused by my Linux system version, which is the version of GLIBC is 2.28, but I can't install GLIBC_2.29 by myself since I use the university's cluster.
Do you know how to solve this problem?

Visualization of novel views

How can I visualize the constructed map/scene from novel viewpoints?

On running python slam.py --config configs/mono/tum/fr3_office.yaml, the results directory has .ply files but I'm not seeing the Gaussians themselves that represent the scene being stored anywhere.

rasterization result is strange.

gaussian_renderer.py

line 130

rendered_image, radii, depth, opacity, n_touched = rasterizer(
means3D=means3D,
means2D=means2D,
shs=shs,
colors_precomp=colors_precomp,
opacities=opacity,
scales=scales,
rotations=rotations,
cov3D_precomp=cov3D_precomp,
theta=viewpoint_camera.cam_rot_delta,
rho=viewpoint_camera.cam_trans_delta,
)

rendered_image and depth, opacity shape is same for original image
but, my result is below;

rendered_image
tensor([[[0.]],

        [[0.]],

        [[0.]]], device='cuda:0')

depth
tensor([[[0.]]], device='cuda:0')
opacity
tensor([[[0.]]], device='cuda:0')

Can you tell me in what situation this happens?

building "diff-gaussian-rasterization" error: expected an expression detected during instantiation of

Thanks for the great work. When I try to create the project env using Win10,I found this wrong :
D:\MonoGS\submodules\diff-gaussian-rasterization\cuda_rasterizer\auxiliary.h(151): warning #177-D: variable "p_proj" was declared but never referenced

D:\MonoGS\submodules\diff-gaussian-rasterization\cuda_rasterizer\backward.cu(471): warning #177-D: variable "c" was declared but never referenced
detected during instantiation of "void preprocessCUDA(int, int, int, const float3 *, const int *, const float *, const __nv_bool *, const glm::vec3 *, const glm::vec4 *, float, const float *, const float *, const float *, const glm::vec3 *, const float3 *, glm::vec3 *, float *, float *, float *, float *, glm::vec3 *, glm::vec4 *, float *) [with C=3]"
(838): here

D:\MonoGS\submodules\diff-gaussian-rasterization\cuda_rasterizer\backward.cu(472): warning #177-D: variable "d" was declared but never referenced
detected during instantiation of "void preprocessCUDA(int, int, int, const float3 *, const int *, const float *, const __nv_bool *, const glm::vec3 *, const glm::vec4 *, float, const float *, const float *, const float *, const glm::vec3 *, const float3 *, glm::vec3 *, float *, float *, float *, float *, glm::vec3 *, glm::vec4 *, float *) [with C=3]"
(838): here

D:\MonoGS\submodules\diff-gaussian-rasterization\cuda_rasterizer\backward.cu(554): error: expected an expression
detected during instantiation of "void renderCUDA(const uint2 *, const uint32_t *, int, int, const float *, const float2 *, const float4 *, const float *, const float *, const float *, const uint32_t *, const float *, const float *, float3 *, float4 *, float *, float *, float *) [with C=3U]"
(882): here

How to increase the frequency?

Hi, thanks for your excellent work!
The paper shows that on TUM fr3/office dataset, the FPS is 2.5 hz and Total Time is 986.7s.
But my test result is approximately 1.4 hz.
I test on with GeForce RTX 4090 and 3060ti respectively for many times and i am sure that the cuda is not out of memory.
I set "use_gui=False" and "single_thread=False" in config yaml file.
Which flag should i set to reproduce the FPS in paper?

The following are my test results for several times:

-------------------------------
Eval: RMSE ATE [m] 0.016370226803308005
Eval: Total time 1779.5885
Eval: Total FPS 1.4132480626841541 
--------------------------------
Eval: RMSE ATE [m] 0.016595806614821122
Eval: Total time 1743.092375
Eval: Total FPS 1.4428380480982828
--------------------------------
Eval: RMSE ATE [m] 0.015768798612186292
Eval: Total time 1718.97075
Eval: Total FPS 1.4630848139795283
--------------------------------
Eval: RMSE ATE [m] 0.01636992988959541
Eval: Total time 1944.81725
Eval: Total FPS 1.2931806317534462
--------------------------------

pip install submodules/simple-knn wrong

Thanks for the great work. When I try to create the project env using Win10,I found this wrong :OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.

(MonoGS) PS D:\MonoGS> pip install submodules/simple-knn
Looking in indexes: https://mirrors.aliyun.com/pypi/simple/
Processing d:\monogs\submodules\simple-knn
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [12 lines of output]
      Traceback (most recent call last):
        File "<string>", line 36, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "D:\MonoGS\submodules\simple-knn\setup.py", line 30, in <module>
          extra_compile_args={"nvcc": [], "cxx": cxx_compiler_flags})
        File "D:\Conda\envs\MonoGS\lib\site-packages\torch\utils\cpp_extension.py", line 983, in CUDAExtension
          library_dirs += library_paths(cuda=True)
        File "D:\Conda\envs\MonoGS\lib\site-packages\torch\utils\cpp_extension.py", line 1105, in library_paths
          paths.append(_join_cuda_home(lib_dir))
        File "D:\Conda\envs\MonoGS\lib\site-packages\torch\utils\cpp_extension.py", line 2125, in _join_cuda_home
          raise EnvironmentError('CUDA_HOME environment variable is not set. '
      OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Torch matrix inversion error

I've (almost) gotten the repo working on Windows with the help of Issue 16.

When I run on the sample data (even using --eval) I hit this issue:
torch._C._LinAlgError: torch.linalg.inv: The diagonal element 1 is zero, the inversion could not be completed because the input matrix is singular.

python slam.py --config configs/mono/tum/fr3_office.yaml --eval

MonoGS: Running MonoGS in Evaluation Mode
MonoGS: Following config will be overriden
MonoGS:         save_results=True
MonoGS:         use_gui=False
MonoGS:         eval_rendering=True
MonoGS:         use_wandb=True
MonoGS: saving results in results\datasets_tum\2024-03-12-13-18-30
wandb: (1) Create a W&B account
wandb: (2) Use an existing W&B account
wandb: (3) Don't visualize my results
wandb: Enter your choice: 3
wandb: You chose "Don't visualize my results"
wandb: Tracking run with wandb version 0.16.4
wandb: W&B syncing is set to `offline` in this directory.
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
MonoGS: Resetting the system
MonoGS: Initialized map
Process Process-3:
Traceback (most recent call last):
  File "C:\Users\Tavius\miniconda3\envs\MonoGS\lib\multiprocessing\process.py", line 297, in _bootstrap
    self.run()
  File "C:\Users\Tavius\miniconda3\envs\MonoGS\lib\multiprocessing\process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "X:\Projects\_2024\MonoGS\utils\slam_backend.py", line 417, in run
    self.add_next_kf(cur_frame_idx, viewpoint, depth_map=depth_map)
  File "X:\Projects\_2024\MonoGS\utils\slam_backend.py", line 69, in add_next_kf
    viewpoint, kf_id=frame_idx, init=init, scale=scale, depthmap=depth_map
  File "X:\Projects\_2024\MonoGS\gaussian_splatting\scene\gaussian_model.py", line 239, in extend_from_pcd_seq
    self.create_pcd_from_image(cam_info, init, scale=scale, depthmap=depthmap)
  File "X:\Projects\_2024\MonoGS\gaussian_splatting\scene\gaussian_model.py", line 131, in create_pcd_from_image
    return self.create_pcd_from_image_and_depth(cam, rgb, depth, init)
  File "X:\Projects\_2024\MonoGS\gaussian_splatting\scene\gaussian_model.py", line 150, in create_pcd_from_image_and_depth
    W2C = getWorld2View2(cam.R, cam.T).cpu().numpy()
  File "X:\Projects\_2024\MonoGS\gaussian_splatting\utils\graphics_utils.py", line 41, in getWorld2View2
    C2W = torch.linalg.inv(Rt)
torch._C._LinAlgError: torch.linalg.inv: The diagonal element 1 is zero, the inversion could not be completed because the input matrix is singular.
[W C:\cb\pytorch_1000000000000\work\torch\csrc\CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]

This is a fresh install using --recursive and only incorporating the change noted above.

How to resolve the issue of the visualization interface not displaying?

Great job! My system is Ubuntu 20.04, and all the necessary Python packages are installed without any other issues. However, the visualization interface is not showing up. How can I solve this problem?

results/tum_datasets_rgbd_dataset_freiburg3_long_office_household/2024-03-06-
10-57-12
FEngine (64 bits) created at 0x381d9f50 (threading is enabled)
FEngine resolved backend: OpenGL
Process Process-4:
Traceback (most recent call last):
  File "/home/sirius/anaconda3/envs/MonoGS/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/sirius/anaconda3/envs/MonoGS/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/data/chenshilang_files/MonoGS/gui/slam_gui.py", line 687, in run
    win = SLAM_GUI(params_gui)
  File "/data/chenshilang_files/MonoGS/gui/slam_gui.py", line 66, in __init__
    self.g_renderer = OpenGLRenderer(self.g_camera.w, self.g_camera.h)
  File "/data/chenshilang_files/MonoGS/gui/gl_render/render_ogl.py", line 71, in __init__
    os.path.join(cur_path, "shaders/gau_frag.glsl"),
  File "/data/chenshilang_files/MonoGS/gui/gl_render/util.py", line 90, in load_shaders
    shaders.compileShader(vertex_shader, GL_VERTEX_SHADER),
  File "/home/sirius/anaconda3/envs/MonoGS/lib/python3.7/site-packages/OpenGL/GL/shaders.py", line 226, in compileShader
    shaderType,
RuntimeError: ("Shader compile failure (0): b'0:1(10): error: GLSL 4.30 is not supported. Supported versions are: 1.10, 1.20, 1.30, 1.40, 1.00 ES, and 3.00 ES\\n'", [b'#version 430 core\n\n#define SH_C0 0.28209479177387814f\n#define SH_C1 0.4886025119029199f\n\n#define SH_C2_0 1.0925484305920792f\n#define SH_C2_1 -1.0925484305920792f\n#define SH_C2_2 0.31539156525252005f\n#define SH_C2_3 -1.0925484305920792f\n#define SH_C2_4 0.5462742152960396f\n\n#define SH_C3_0 -0.5900435899266435f\n#define SH_C3_1 2.890611442640554f\n#define SH_C3_2 -0.4570457994644658f\n#define SH_C3_3 0.3731763325901154f\n#define SH_C3_4 -0.4570457994644658f\n#define SH_C3_5 1.445305721320277f\n#define SH_C3_6 -0.5900435899266435f\n\nlayout(location = 0) in vec2 position;\n// layout(location = 1) in vec3 g_pos;\n// layout(location = 2) in vec4 g_rot;\n// layout(location = 3) in vec3 g_scale;\n// layout(location = 4) in vec3 g_dc_color;\n// layout(location = 5) in float g_opacity;\n\n\n#define POS_IDX 0\n#define ROT_IDX 3\n#define SCALE_IDX 7\n#define OPACITY_IDX 10\n#define SH_IDX 11\n\nlayout (std430, binding=0) buffer gaussian_data {\n\tfloat g_data[];\n\t// compact version of following data\n\t// vec3 g_pos[];\n\t// vec4 g_rot[];\n\t// vec3 g_scale[];\n\t// float g_opacity[];\n\t// vec3 g_sh[];\n};\nlayout (std430, binding=1) buffer gaussian_order {\n\tint gi[];\n};\n\nuniform mat4 view_matrix;\nuniform mat4 projection_matrix;\nuniform vec3 hfovxy_focal;\nuniform vec3 cam_pos;\nuniform int sh_dim;\nuniform float scale_modifier;\nuniform int render_mod;  // > 0 render 0-ith SH dim, -1 depth, -2 bill board, -3 gaussian\n\nout vec3 color;\nout float alpha;\nout vec3 conic;\nout vec2 coordxy;  // local coordinate in quad, unit in pixel\n\nmat3 computeCov3D(vec3 scale, vec4 q)  // should be correct\n{\n    mat3 S = mat3(0.f);\n    S[0][0] = scale.x;\n\tS[1][1] = scale.y;\n\tS[2][2] = scale.z;\n\tfloat r = q.x;\n\tfloat x = q.y;\n\tfloat y = q.z;\n\tfloat z = q.w;\n\n    mat3 R = mat3(\n\t\t1.f - 2.f * (y * y + z * z), 2.f * (x * y - r * z), 2.f * (x * z + r * y),\n\t\t2.f * (x * y + r * z), 1.f - 2.f * (x * x + z * z), 2.f * (y * z - r * x),\n\t\t2.f * (x * z - r * y), 2.f * (y * z + r * x), 1.f - 2.f * (x * x + y * y)\n\t);\n\n    mat3 M = S * R;\n    mat3 Sigma = transpose(M) * M;\n    return Sigma;\n}\n\nvec3 computeCov2D(vec4 mean_view, float focal_x, float focal_y, float tan_fovx, float tan_fovy, mat3 cov3D, mat4 viewmatrix)\n{\n    vec4 t = mean_view;\n    // why need this? Try remove this later\n    float limx = 1.3f * tan_fovx;\n    float limy = 1.3f * tan_fovy;\n    float txtz = t.x / t.z;\n    float tytz = t.y / t.z;\n    t.x = min(limx, max(-limx, txtz)) * t.z;\n    t.y = min(limy, max(-limy, tytz)) * t.z;\n\n    mat3 J = mat3(\n        focal_x / t.z, 0.0f, -(focal_x * t.x) / (t.z * t.z),\n\t\t0.0f, focal_y / t.z, -(focal_y * t.y) / (t.z * t.z),\n\t\t0, 0, 0\n    );\n    mat3 W = transpose(mat3(viewmatrix));\n    mat3 T = W * J;\n\n    mat3 cov = transpose(T) * transpose(cov3D) * T;\n    // Apply low-pass filter: every Gaussian should be at least\n\t// one pixel wide/high. Discard 3rd row and column.\n\tcov[0][0] += 0.3f;\n\tcov[1][1] += 0.3f;\n    return vec3(cov[0][0], cov[0][1], cov[1][1]);\n}\n\nvec3 get_vec3(int offset)\n{\n\treturn vec3(g_data[offset], g_data[offset + 1], g_data[offset + 2]);\n}\nvec4 get_vec4(int offset)\n{\n\treturn vec4(g_data[offset], g_data[offset + 1], g_data[offset + 2], g_data[offset + 3]);\n}\n\nvoid main()\n{\n\tint boxid = gi[gl_InstanceID];\n\tint total_dim = 3 + 4 + 3 + 1 + sh_dim;\n\tint start = boxid * total_dim;\n\tvec4 g_pos = vec4(get_vec3(start + POS_IDX), 1.f);\n    vec4 g_pos_view = view_matrix * g_pos;\n    vec4 g_pos_screen = projection_matrix * g_pos_view;\n\tg_pos_screen.xyz = g_pos_screen.xyz / g_pos_screen.w;\n    g_pos_screen.w = 1.f;\n\t// early culling\n\tif (any(greaterThan(abs(g_pos_screen.xyz), vec3(1.3))))\n\t{\n\t\tgl_Position = vec4(-100, -100, -100, 1);\n\t\treturn;\n\t}\n\tvec4 g_rot = get_vec4(start + ROT_IDX);\n\tvec3 g_scale = get_vec3(start + SCALE_IDX);\n\tfloat g_opacity = g_data[start + OPACITY_IDX];\n\n    mat3 cov3d = computeCov3D(g_scale * scale_modifier, g_rot);\n    vec2 wh = 2 * hfovxy_focal.xy * hfovxy_focal.z;\n    vec3 cov2d = computeCov2D(g_pos_view, \n                              hfovxy_focal.z, \n                              hfovxy_focal.z, \n                              hfovxy_focal.x, \n                              hfovxy_focal.y, \n                              cov3d, \n                              view_matrix);\n\n    // Invert covariance (EWA algorithm)\n\tfloat det = (cov2d.x * cov2d.z - cov2d.y * cov2d.y);\n\tif (det == 0.0f)\n\t\tgl_Position = vec4(0.f, 0.f, 0.f, 0.f);\n    \n    float det_inv = 1.f / det;\n\tconic = vec3(cov2d.z * det_inv, -cov2d.y * det_inv, cov2d.x * det_inv);\n    \n    vec2 quadwh_scr = vec2(3.f * sqrt(cov2d.x), 3.f * sqrt(cov2d.z));  // screen space half quad height and width\n    vec2 quadwh_ndc = quadwh_scr / wh * 2;  // in ndc space\n    g_pos_screen.xy = g_pos_screen.xy + position * quadwh_ndc;\n    coordxy = position * quadwh_scr;\n    gl_Position = g_pos_screen;\n    \n    alpha = g_opacity;\n\n\tif (render_mod == -1)\n\t{\n\t\tfloat depth = -g_pos_view.z;\n\t\tdepth = depth < 0.05 ? 1 : depth;\n\t\tdepth = 1 / depth;\n\t\tcolor = vec3(depth, depth, depth);\n\t\treturn;\n\t}\n\n\t// Covert SH to color\n\tint sh_start = start + SH_IDX;\n\tvec3 dir = g_pos.xyz - cam_pos;\n    dir = normalize(dir);\n\tcolor = SH_C0 * get_vec3(sh_start);\n\t\n\tif (sh_dim > 3 && render_mod >= 1)  // 1 * 3\n\t{\n\t\tfloat x = dir.x;\n\t\tfloat y = dir.y;\n\t\tfloat z = dir.z;\n\t\tcolor = color - SH_C1 * y * get_vec3(sh_start + 1 * 3) + SH_C1 * z * get_vec3(sh_start + 2 * 3) - SH_C1 * x * get_vec3(sh_start + 3 * 3);\n\n\t\tif (sh_dim > 12 && render_mod >= 2)  // (1 + 3) * 3\n\t\t{\n\t\t\tfloat xx = x * x, yy = y * y, zz = z * z;\n\t\t\tfloat xy = x * y, yz = y * z, xz = x * z;\n\t\t\tcolor = color +\n\t\t\t\tSH_C2_0 * xy * get_vec3(sh_start + 4 * 3) +\n\t\t\t\tSH_C2_1 * yz * get_vec3(sh_start + 5 * 3) +\n\t\t\t\tSH_C2_2 * (2.0f * zz - xx - yy) * get_vec3(sh_start + 6 * 3) +\n\t\t\t\tSH_C2_3 * xz * get_vec3(sh_start + 7 * 3) +\n\t\t\t\tSH_C2_4 * (xx - yy) * get_vec3(sh_start + 8 * 3);\n\n\t\t\tif (sh_dim > 27 && render_mod >= 3)  // (1 + 3 + 5) * 3\n\t\t\t{\n\t\t\t\tcolor = color +\n\t\t\t\t\tSH_C3_0 * y * (3.0f * xx - yy) * get_vec3(sh_start + 9 * 3) +\n\t\t\t\t\tSH_C3_1 * xy * z * get_vec3(sh_start + 10 * 3) +\n\t\t\t\t\tSH_C3_2 * y * (4.0f * zz - xx - yy) * get_vec3(sh_start + 11 * 3) +\n\t\t\t\t\tSH_C3_3 * z * (2.0f * zz - 3.0f * xx - 3.0f * yy) * get_vec3(sh_start + 12 * 3) +\n\t\t\t\t\tSH_C3_4 * x * (4.0f * zz - xx - yy) * get_vec3(sh_start + 13 * 3) +\n\t\t\t\t\tSH_C3_5 * z * (xx - yy) * get_vec3(sh_start + 14 * 3) +\n\t\t\t\t\tSH_C3_6 * x * (xx - 3.0f * yy) * get_vec3(sh_start + 15 * 3);\n\t\t\t}\n\t\t}\n\t}\n\tcolor += 0.5f;\n}\n'], GL_VERTEX_SHADER)

Conda env buildin error - ERROR: Directory 'submodules/simple-knn' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.

While building conda env simple-knn packages is not found. Does anyone face this problem? pip install simple_knn does not help at all.