How to use the Hybrid representation (Mesh + Gaussians on the surface) in 3d softwares

[xiiliu]

How to use the Hybrid representation (Mesh + Gaussians on the surface) in 3d softwares

[Anttwo]

Hello xiiliu,

I just added a short notebook that shows how to render images with the hybrid representation in Python.
However, it's true that there is currently no way to render in a dedicated 3D viewer or Blender the hybrid representation in real-time, as it is a non-trivial engineering task; But I'm working on it! The only limitation was that it was not possible to use the Gaussian Splatting rasterizer in 3D softwares.

However, I saw that there are a few Gaussian Splatting real-time viewers out there, and even Unity, Unreal Engines and Blender extensions. I'm currently taking a look at these, and I will try to propose some viewer or extension to visualize the hybrid representation.

For the presentation video, I first animated the meshes in Blender, then exported the animation files inside python to render with SuGaR. The scripts are inside the ./blender/ directory, and the dedicated class to load the blender camera/animation data inside SuGaR is in ./sugar_scene/sugar_compositor.py. But the script in the repo is not finished yet, since I'm still cleaning the dev code for that.

I will push the final version of this script and provide a tutorial as soon as possible.

[xiiliu]

Thank you for the reply and I got your point.

Yes, there are already some plugins specifically designed for Gaussian Splatting available in Unity or UE, and I have used them. With the help of these plugins, we can do more things in industry. Like this one for Unity: https://github.com/aras-p/UnityGaussianSplatting

But since the rendering way of SuGaR is different with Gaussian Splatting, so the plugins don't work with SuGaR. Hope SuGaR can has its own plugins in 3D softwares soon~