anttwo/sugar

[CVPR 2024] Official PyTorch implementation of SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering

sugar's Issues

./outputcameras.json missing | Getting error in second step after Saving Gaussians and Eval Train

load_gs_cameras
with open(gs_output_path + 'cameras.json') as f:
FileNotFoundError: [Errno 2] No such file or directory: './outputcameras.json

python train.py -s ./data/playroom -c ./output -r sdf

cameras.json is in the output path. But I cannot find ./outputcameras.json as well.

I have implemented all regularizations, anybody isis interested in same surface-recon and discuss with me?

I urgently need a partner for discussion who shares the same direction.

Thanks for the authors of SuGaR for releasing the code. learning ...

great

To pay tribute to SuGaR authors, reproduce SuGaR algorithm in code!

1. regularization: guass-points and gauss-point surface-alignment regularization item(s)
   'read' the paper line-by-line, char-by-char
   'draw' my learning-note as below
   'code' the depth-map (the SuGaR author mentioned solution#1, not SuGaR picked solution#2) (speedup^2 paper Eq.9)
   'code' the MultivariateNormal sampling (paper Eq.9) (as below)
   'code' sampling p
   'code' argmin
   ‘code' all 2023-12-18
   In Chinese: 有点担心哪里理解的不对
   here: #17
   DONE

2. extraction: mesh-extraction by Poisson reconstruction, not Marching-Cubes
   TODO

3. joint: joint-optimization both official GS and SuGaR-constraints
   TODO

[Open3D WARNING] Read PLY failed: unable to open file:

Greetings from Canada Antoine,
I am a filmmaker and have been working with various aspects of Virtual Production since its inception. I am by no means a computer expert, and to some, this may be a simple issue to solve. Everything works great up to the point of "optimize a SuGaR model".

Loading mesh to bind to:
./output/coarse_mesh\ball2\sugarmesh_coarse\ball2\3Dgs7000_sdfestim02_sdfnorm02_level03_decim1000000.ply...
RPly: Unable to open file
[Open3D WARNING] Read PLY failed: unable to open file: ./output/coarse_mesh\ball2\sugarmesh_coarse\ball2\3Dgs7000_sdfestim02_sdfnorm02_level03_decim1000000.ply
Mesh to bind to loaded.
Binding radiance cloud to surface mesh...
Traceback (most recent call last):
File "C:\Users\conan\SuGaR\train.py", line 160, in
refined_sugar_path = refined_training(refined_args)
File "C:\Users\conan\SuGaR\sugar_trainers\refine.py", line 393, in refined_training
sugar = SuGaR(
File "C:\Users\conan\SuGaR\sugar_scene\sugar_model.py", line 322, in init
scales = scales.clamp_min(0.0000001).reshape(len(faces_verts), -1, 1).expand(-1, self.n_gaussians_per_surface_triangle, 2).clone().reshape(-1, 2)
RuntimeError: cannot reshape tensor of 0 elements into shape [0, -1, 1] because the unspecified dimension size -1 can be any value and is ambiguous

Merci pour le travail que vous faites. Je pense que cela pourrait changer le monde du cinéma.
Best,
Kit

哎，搞了半天，人们还是需要深度，需要精确的表面。

GS本质是放松了对一个物体重构的约束，就是视觉上已经很像了哈，但那个高斯点虽然与shape有关，但代表不了表面甚至差很远。

然后呢，在游戏，虚拟现实中，只要需要“interactive”，不只是“see see”的，还是需要mesh这种和物理现实逼近的表面。

SuGaR等还是得继续对GS加强约束， depth啊，normals啊，那啥啥啥的，还是得直接或者间接的得到和利用。

I am using Windows 10 and have some issues

@Anttwo
Hello Antoine!
I am using Windows 10 and I know of the issues with path (/ and ) with the differences between Linux and Windows conventions.
Do you foresee having those changes made soon or would you suggest that I use a different operating system in the meantime, meaning Ubuntu?

Also have:
128 GB Ram
RTX 3060 12 GB

The RTX 3060 is a hindrance as I have so many Photogrammetry models I have amassed over the years that I would love to use. Some of which are well over 750 photos.
With my setup, I am limited to under 75 since the train.py takes an enormous amount of time with these limitations. What do you suggest as a minimum GPU for the use of up to and over 1000 photos perhaps up to 2000?

Here is my progress so far for all to see as I am sure many people are having the same problem:

Test run dec21 2023

CONVERT

(sugar) C:\Users\conan\SuGaR>python gaussian_splatting/convert.py -s data/bullet_01

....Undistorting image [50/50]
Writing reconstruction...
Writing configuration...
Writing scripts...
Elapsed time: 0.034 [minutes]
Done.

TRAIN

(sugar) C:\Users\conan\SuGaR>python gaussian_splatting/train.py -s data/bullet_01 -m output/bullet_01 --iterations 30000 --save_iterations 7000 30000 --checkpoint_iterations 7000 30000

...[ITER 30000] Evaluating train: L1 0.013728742301464082 PSNR 33.831542587280275 [21/12 11:57:32]

[ITER 30000] Saving Gaussians [21/12 11:57:32]

[ITER 30000] Saving Checkpoint [21/12 11:57:55]

Training complete. [21/12 11:57:59]

OPTIMIZE

(sugar) C:\Users\conan\SuGaR>python train.py -s data/bullet_01 -c output/bullet_01/ -r "density"

...Loading mesh to bind to:
./output/coarse_mesh\bullet_01\sugarmesh_coarse\bullet_01\3Dgs7000_densityestim02_sdfnorm02_level03_decim1000000.ply...
RPly: Unable to open file
[Open3D WARNING] Read PLY failed: unable to open file: ./output/coarse_mesh\bullet_01\sugarmesh_coarse\bullet_01\3Dgs7000_densityestim02_sdfnorm02_level03_decim1000000.ply
Mesh to bind to loaded.
Binding radiance cloud to surface mesh...
Traceback (most recent call last):
File "C:\Users\conan\SuGaR\train.py", line 160, in
refined_sugar_path = refined_training(refined_args)
File "C:\Users\conan\SuGaR\sugar_trainers\refine.py", line 393, in refined_training
sugar = SuGaR(
File "C:\Users\conan\SuGaR\sugar_scene\sugar_model.py", line 322, in init
scales = scales.clamp_min(0.0000001).reshape(len(faces_verts), -1, 1).expand(-1, self.n_gaussians_per_surface_triangle, 2).clone().reshape(-1, 2)
RuntimeError: cannot reshape tensor of 0 elements into shape [0, -1, 1] because the unspecified dimension size -1 can be any value and is ambiguous

Thanks for your work!

OOM

I get OOM in that particular section (texturing) when trying to run --high_poly, I disabled Pytorch CUDA memory caching to see if it would mitigate but it doesn't solve it. I Am running Ubuntu 22.04, RTX 4090 16GB VRAM and 32GB of RAM. That seems to not be enough? Any pointers would be greatly appreciated, Here are the parameters I run the processing with:

Training the Gaussian:

gaussian_splatting$ python train.py -s /home/xxx/SuGaR/xxx/xxxxxxxxxxxx --iterations 7000 -m /home/xxx/SuGaR/xxxxxx/xxxxxxxxxx

Rest of the steps:

python train.py -s /home/xxx/SuGaR/xxxx/xxxxxxxxxxxx -c /home/reyxxxSuGaR/xxxxxx/xxxxxxxxxx -r "sdf" --high_poly True --refinement_time "medium"

Here is the output:

Number of gaussians per surface triangle: 1
Square size: 10
Postprocess mesh: False

Source path: /home/xxx/SuGaR/xx/xxxxxxxxxx/
Gaussian splatting checkpoint path: /home/xx/SuGaR/xxxxxx/xxxxxxxxxx/

Loading Vanilla 3DGS model config /home/xxx/SuGaR/xxxxxx/xxxxxxxxxx/...
Found image extension .png
Vanilla 3DGS Loaded.
22 training images detected.
The model has been trained for 7000 steps.
0.870508 M gaussians detected.
Binding radiance cloud to surface mesh...
Traceback (most recent call last):
File "/home/xx/SuGaR/train.py", line 180, in
refined_mesh_path = extract_mesh_and_texture_from_refined_sugar(refined_mesh_args)
File "/home/xx/SuGaR/sugar_extractors/refined_mesh.py", line 193, in extract_mesh_and_texture_from_refined_sugar
verts_uv, faces_uv, texture_img = extract_texture_image_and_uv_from_gaussians(
File "/home/xxx/SuGaR/sugar_scene/sugar_model.py", line 2420, in extract_texture_image_and_uv_from_gaussians
texture_img = SH2RGB(texture_img.flip(0))
File "/home/xxx/SuGaR/sugar_utils/spherical_harmonics.py", line 178, in SH2RGB
return sh * C0 + 0.5
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Questions about the equations in the paper

Hi, Thanks for uploading this awesome work, the result is very cool and interesting!

I have few maybe naive questions about the equations in the paper and hope for a discussion.

• I am not fully understanding why eq. (4) hold?
• For eq. (6), my high-level understanding is that instead of opacity, you choose to regularize SDF, so eq.(6) serves as a conversion from opacity to SDF? How do you come to this particular formulation?
• For Fig.5, why can we compute f(p) by computing the difference between the depth of p's projection and the true depth of p?

Looking forward your reply!

mamba env create -f environment.yml freaks out W11

Create work! overall!!!

Tried to install the environment from scratch and it throws errors, like hell.
Does anyone has the same issue? Or know how to fix this? Do not want to type everything by hand.

`Looking for: ['_libgcc_mutex==0.1=main', '_openmp_mutex==5.1=1_gnu', 'anyio==4.1.0=pyhd8ed1ab_0', 'argon2-cffi==21.1.0=]

Could not solve for environment specs
The following packages are incompatible
├─ _openmp_mutex ==5.1 1_gnu does not exist (perhaps a typo or a missing channel);
├─ argon2-cffi ==21.1.0 py39h3811e60_2 does not exist (perhaps a typo or a missing channel);
├─ brotli-python ==1.0.9 py39h6a678d5_7 does not exist (perhaps a typo or a missing channel);
├─ bzip2 ==1.0.8 h7b6447c_0 does not exist (perhaps a typo or a missing channel);
├─ ca-certificates ==2023.11.17 hbcca054_0 does not exist (perhaps a typo or a missing channel);
├─ cffi ==1.16.0 py39h5eee18b_0 does not exist (perhaps a typo or a missing channel);
├─ cryptography ==41.0.7 py39hdda0065_0 does not exist (perhaps a typo or a missing channel);
├─ ffmpeg ==4.3 hf484d3e_0 does not exist (perhaps a typo or a missing channel);
├─ filelock ==3.13.1 py39h06a4308_0 does not exist (perhaps a typo or a missing channel);
├─ freetype ==2.12.1 h4a9f257_0 does not exist (perhaps a typo or a missing channel);
├─ giflib ==5.2.1 h5eee18b_3 does not exist (perhaps a typo or a missing channel);
├─ gmp ==6.2.1 h295c915_3 does not exist (perhaps a typo or a missing channel);
├─ gmpy2 ==2.1.2 py39heeb90bb_0 does not exist (perhaps a typo or a missing channel);
├─ gnutls ==3.6.15 he1e5248_0 does not exist (perhaps a typo or a missing channel);
├─ idna ==3.4 py39h06a4308_0 does not exist (perhaps a typo or a missing channel);
├─ intel-openmp ==2023.1.0 hdb19cb5_46306 does not exist (perhaps a typo or a missing channel);
├─ ipykernel ==5.5.5 py39hef51801_0 does not exist (perhaps a typo or a missing channel);
├─ ipython ==8.18.1 pyh707e725_3 is not installable because it requires
│ └─ __unix, which is missing on the system;
├─ jinja2 ==3.1.2 py39h06a4308_0 does not exist (perhaps a typo or a missing channel);
├─ jpeg ==9e h5eee18b_1 does not exist (perhaps a typo or a missing channel);
├─ jsonpointer ==2.4 py39hf3d152e_3 does not exist (perhaps a typo or a missing channel);
├─ jupyter_core ==5.5.0 py39hf3d152e_0 does not exist (perhaps a typo or a missing channel);
├─ lame ==3.100 h7b6447c_0 does not exist (perhaps a typo or a missing channel);
├─ lcms2 ==2.12 h3be6417_0 does not exist (perhaps a typo or a missing channel);
├─ ld_impl_linux-64 ==2.38 h1181459_1 does not exist (perhaps a typo or a missing channel);
├─ lerc ==3.0 h295c915_0 does not exist (perhaps a typo or a missing channel);
├─ libcufile ==1.8.1.2 0 does not exist (perhaps a typo or a missing channel);
├─ libdeflate ==1.17 h5eee18b_1 does not exist (perhaps a typo or a missing channel);
├─ libffi ==3.4.4 h6a678d5_0 does not exist (perhaps a typo or a missing channel);
├─ libgcc-ng ==11.2.0 h1234567_1 does not exist (perhaps a typo or a missing channel);
├─ libgcc ==7.2.0 h69d50b8_2 does not exist (perhaps a typo or a missing channel);
├─ libgomp ==11.2.0 h1234567_1 does not exist (perhaps a typo or a missing channel);
├─ libiconv ==1.16 h7f8727e_2 does not exist (perhaps a typo or a missing channel);
├─ libidn2 ==2.3.4 h5eee18b_0 does not exist (perhaps a typo or a missing channel);
├─ libpng ==1.6.39 h5eee18b_0 does not exist (perhaps a typo or a missing channel);
├─ libsodium ==1.0.18 h36c2ea0_1 does not exist (perhaps a typo or a missing channel);
├─ libstdcxx-ng ==11.2.0 h1234567_1 does not exist (perhaps a typo or a missing channel);
├─ libtasn1 ==4.19.0 h5eee18b_0 does not exist (perhaps a typo or a missing channel);
├─ libtiff ==4.5.1 h6a678d5_0 does not exist (perhaps a typo or a missing channel);
├─ libunistring ==0.9.10 h27cfd23_0 does not exist (perhaps a typo or a missing channel);
├─ libwebp-base ==1.3.2 h5eee18b_0 does not exist (perhaps a typo or a missing channel);
├─ libwebp ==1.3.2 h11a3e52_0 does not exist (perhaps a typo or a missing channel);
├─ lz4-c ==1.9.4 h6a678d5_0 does not exist (perhaps a typo or a missing channel);
├─ markupsafe ==2.1.1 py39h7f8727e_0 does not exist (perhaps a typo or a missing channel);
├─ mkl-service ==2.4.0 py39h5eee18b_1 does not exist (perhaps a typo or a missing channel);
├─ mkl ==2023.1.0 h213fc3f_46344 does not exist (perhaps a typo or a missing channel);
├─ mkl_fft ==1.3.8 py39h5eee18b_0 does not exist (perhaps a typo or a missing channel);
├─ mkl_random ==1.2.4 py39hdb19cb5_0 does not exist (perhaps a typo or a missing channel);
├─ mpc ==1.1.0 h10f8cd9_1 does not exist (perhaps a typo or a missing channel);
├─ mpfr ==4.0.2 hb69a4c5_1 does not exist (perhaps a typo or a missing channel);
├─ mpmath ==1.3.0 py39h06a4308_0 does not exist (perhaps a typo or a missing channel);
├─ ncurses ==6.4 h6a678d5_0 does not exist (perhaps a typo or a missing channel);
├─ nettle ==3.7.3 hbbd107a_1 does not exist (perhaps a typo or a missing channel);
├─ networkx ==3.1 py39h06a4308_0 does not exist (perhaps a typo or a missing channel);
├─ numpy-base ==1.26.2 py39hb5e798b_0 does not exist (perhaps a typo or a missing channel);
├─ numpy ==1.26.2 py39h5f9d8c6_0 does not exist (perhaps a typo or a missing channel);
├─ openh264 ==2.1.1 h4ff587b_0 does not exist (perhaps a typo or a missing channel);
├─ openjpeg ==2.4.0 h3ad879b_0 does not exist (perhaps a typo or a missing channel);
├─ openssl ==3.0.12 h7f8727e_0 does not exist (perhaps a typo or a missing channel);
├─ pillow ==10.0.1 py39ha6cbd5a_0 does not exist (perhaps a typo or a missing channel);
├─ pip ==23.3.1 py39h06a4308_0 does not exist (perhaps a typo or a missing channel);
├─ portalocker ==2.8.2 py39hf3d152e_1 does not exist (perhaps a typo or a missing channel);
├─ pyopenssl ==23.2.0 py39h06a4308_0 does not exist (perhaps a typo or a missing channel);
├─ pysocks ==1.7.1 py39h06a4308_0 does not exist (perhaps a typo or a missing channel);
├─ python ==3.9.18 h955ad1f_0 does not exist (perhaps a typo or a missing channel);
├─ pytorch-cuda ==11.8 h7e8668a_5 does not exist (perhaps a typo or a missing channel);
├─ pytorch3d ==0.7.4 py39_cu118_pyt201 does not exist (perhaps a typo or a missing channel);
├─ pytorch ==2.0.1 py3.9_cuda11.8_cudnn8.7.0_0 does not exist (perhaps a typo or a missing channel);
├─ pyyaml ==6.0 py39hb9d737c_4 does not exist (perhaps a typo or a missing channel);
├─ pyzmq ==25.1.0 py39h6a678d5_0 does not exist (perhaps a typo or a missing channel);
├─ readline ==8.2 h5eee18b_0 does not exist (perhaps a typo or a missing channel);
├─ requests ==2.31.0 py39h06a4308_0 does not exist (perhaps a typo or a missing channel);
├─ send2trash ==1.8.2 pyh41d4057_0 is not installable because it requires
│ └─ __linux, which is missing on the system;
├─ setuptools ==68.2.2 py39h06a4308_0 does not exist (perhaps a typo or a missing channel);
├─ sqlite ==3.41.2 h5eee18b_0 does not exist (perhaps a typo or a missing channel);
├─ sympy ==1.12 py39h06a4308_0 does not exist (perhaps a typo or a missing channel);
├─ tbb ==2021.8.0 hdb19cb5_0 does not exist (perhaps a typo or a missing channel);
├─ tenacity ==8.2.2 py39h06a4308_0 does not exist (perhaps a typo or a missing channel);
├─ terminado ==0.18.0 pyh0d859eb_0 is not installable because it requires
│ └─ __linux, which is missing on the system;
├─ tk ==8.6.12 h1ccaba5_0 does not exist (perhaps a typo or a missing channel);
├─ torchtriton ==2.0.0 py39 does not exist (perhaps a typo or a missing channel);
├─ tornado ==6.3.3 py39h5eee18b_0 does not exist (perhaps a typo or a missing channel);
├─ typing_extensions ==4.7.1 py39h06a4308_0 does not exist (perhaps a typo or a missing channel);
├─ urllib3 ==1.26.18 py39h06a4308_0 does not exist (perhaps a typo or a missing channel);
├─ wheel ==0.41.2 py39h06a4308_0 does not exist (perhaps a typo or a missing channel);
├─ xz ==5.4.5 h5eee18b_0 does not exist (perhaps a typo or a missing channel);
├─ yaml ==0.2.5 h7f98852_2 does not exist (perhaps a typo or a missing channel);
├─ zeromq ==4.3.4 h2531618_0 does not exist (perhaps a typo or a missing channel);
├─ zlib ==1.2.13 h5eee18b_0 does not exist (perhaps a typo or a missing channel);
└─ zstd ==1.5.5 hc292b87_0 does not exist (perhaps a typo or a missing channel).`

Are we on the right path for surface reconstruction?

我们是否走在正确的表面重建路上？ GS本质是放松了“深度/法线/表面/形态/物理实在”的约束，追求了一种2.5D的“视觉上很象”“立体上未必象”“但能够支持观察点立体移动”的准重构。 除了2023-11的法国大神的SuGaR，还有2023-12的新加坡国立的大神的NeuSG，等等，都视图在做到表面的重构。SuGaR的方法个人感觉还是很重，从多变量正态分布采样开始，虽然有利用深度图的一些加速策略，到多正则项。 NeuSG的做法，看起来要轻量级一些。 当然，既然要重构mesh，最好就是达到商业产品可以用的“精美”的质量，形态准确，细节丰富。 一边学习一边反思：GS为啥快，如果最终目标是real 3D mesh而不是2.5D的视觉很象，我们从GS学习啥？和直接mesh-fit到底为啥不容易收敛, GS是因为先放弃了mesh的边/面的约束，让点自由的增山和移动吗？还是用GS点来表示本身就更合适？还是直接用梯度来驱动这些操作很有效？mesh也支持顶点的增删和移动呢？ 等等等等问题， 同理，SDF/UDF等等那么多方向，该从GS核心学习什么？ 所以，假设要沿GS往下走，GS++方向是这样各种加正则项呢，还是增加对一些学习的参数项目，特别关于depth, normals还有强相关的opacity的处理逻辑。 “一个更纯粹更原生的GS++ for mesh”应该是什么样子？

chatgpt translated:
Are we on the right path for surface reconstruction? The essence of GS lies in relaxing constraints such as "depth/normal/surface/form/physical reality," pursuing a quasi-reconstruction that is 2.5D—visually similar, not necessarily volumetrically similar, yet capable of supporting stereoscopic movement of the observation point. In addition to SuGaR from the genius in France in November 2023, there is also NeuSG from the genius at the National University of Singapore in December 2023, and so on—all attempting to achieve surface reconstruction. Personally, SuGaR's method feels quite complex, starting from sampling a multivariate normal distribution, although there are some acceleration strategies using depth maps, leading to multiple regularization terms. NeuSG's approach seems to be more lightweight.

Of course, since we are reconstructing the mesh, it is preferable to achieve "exquisite" quality suitable for commercial products—accurate form with rich details. While learning, there is ongoing reflection: why is GS fast? If the ultimate goal is a real 3D mesh rather than a visually similar 2.5D, what do we learn from GS? Why is it not easy to converge by directly mesh-fitting? Did GS give up constraints on mesh edges/faces, allowing points to freely increase and move? Is it more appropriate to represent GS points themselves, or is it effective to drive these operations directly with gradients? Does the mesh also support adding, deleting, and moving vertices?

And so on, and similarly in various directions such as SDF/UDF—what should we learn from the core of GS? Assuming we continue down the GS path, should GS++ involve adding various regularization terms, or should it increase support for some learning parameters, especially regarding the logic of handling depth, normals, and strongly related opacity? What would a "purer and more native GS++ for mesh" look like?

different mesh way(s), any comment?

graphdeco-inria/gaussian-splatting#540

copy and paste:
"
add regulirization items, force: a) algin points to surface, b)flatten every point。 (refer: SuGaR https://arxiv.org/abs/2311.12775)

learn a mapping fucction, mesh = f_gs_to_mesh(gs_points_convergent, [cameras_in_trainset, target_images_in_trainset])
on this direction, I suppose we can learn the regular pattern(intuitively, there should be), that is better than a hard-code algorithm(maybe difficult) which do the mapping.i
"

encountering error at coarse mesh extraction

Hi, I encountered the following error when I was trying to run the second stage of your code, here is the error message.

Could you kindly give me some suggestions to fix this problem? Thanks!

To be clear, I use the Nerf synthetic dataset and generate the gs checkpoint using your code.

Traceback (most recent call last):
File "train.py", line 136, in
coarse_mesh_args = AttrDict({
File "SuGaR-main/sugar_extractors/coarse_mesh.py", line 410, in extract_mesh_from_coarse_sugar
o3d_bg_mesh, o3d_bg_densities = o3d.geometry.TriangleMesh.create_from_point_cloud_poisson(
RuntimeError: [Open3D Error] (static std::tuple<std::shared_ptropen3d::geometry::TriangleMesh, std::vector<double, std::allocator > > open3d::geometry::TriangleMesh::CreateFromPointCloudPoisson(const open3d::geometry::PointCloud&, size_t, float, float, bool, int)) /root/Open3D/cpp/open3d/geometry/SurfaceReconstructionPoisson.cpp:732: Point cloud has no normals

About calculation of depth map

Thanks for the awesome work!

Could you please provide more details about how to compute the depth map using the Gaussian Splatting rasterizer?

Thank you very much!

To SuGaR-er: the points regularized by SuGaR. very goooooooood!

To SuGaR-er: I can run train_coarse_density.py on 12G GPU (titan-xp), No OOM.

cuda out of memory

Is there any parameter can reduce cuda memory , please?
"cuda out of memory" for 400+ images on 24G cuda device 3090

Questions about using NeRF Synthetic dataset

When trying to use NeRF Synthetic dataset, such as lego, an error [Open3D Error] is reported
(static std::tuple<std::shared_ptropen3d::geometry::TriangleMesh, std::vector<double, std::allocator > > open3d::geometry::TriangleMesh::CreateFromPointCloudPoisson(const open3d::geometry::PointCloud&, size_t, float, float, bool, int)) /root/Open3D/cpp/open3d/geometry/SurfaceReconstructionPoisson.cpp:732 : Point cloud has no normals

console print
Background points:
torch.Size([0, 3])
torch.Size([0, 3])
torch.Size([0, 3])

The lego data have no background, but in the coarse mesh stage, bg_pcd is still processed. Is this a bug?

How to use the Hybrid representation (Mesh + Gaussians on the surface) in 3d softwares

@Anttwo Thanks again for your great work.
Have a question about how to use the Hybrid representation in 3d softwares like Blender, Unity, UE, etc.
After I use 'train.py' to run the entire pipeline, I got the 'refined_mesh' folder with .obj and .png file, these are mesh and texture. I can import them into Blender, but the rendering effect is not satisfactory.
I also get the 'refined' and 'coarse_mesh' folders with .pt files, but I don't know how to use them into Blender.
Could you please give me guidance?

real-case: resin toy wizard reconstruction. all better quality cases are at the end sections! In Chinese: 高斯溅射这条路上提取表面/mesh目前最靠谱的路。好效果的在最后部分。

coarse density:
(the best way to reconstruct surface based on GS, so far. )

one of inputs:

8000 iteations: (from 7000)

sugarmesh_3Dgs7000_densityestim02_sdfnorm02_level03_decim1000000.zip

15000 iterations: (still no refine step)

some better quality cases are in the end sections !!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Interactive Viewers

thanks for your geat work, how to Interactive Viewers with SuGar model pt model ? Looking forward to your reply!
@Anttwo

CUDA OUT OF MEMORY running on a 12gb card

When running mesh extraction, I get a cuda OOM error. Is there anyway to run this method on lower memory cards?

MemoryError: bad allocation when extracting corase mesh

Hi Anttwo！
Thanks for the amazing work and the elegant code!
I ran the project just now, and I encontered the following error just after console printed "Decimating and cleaning meshes".
——————————————————————————————————————————
-----Decimating and cleaning meshes-----

Processing decimation target: 1000000
Decimating foreground mesh...
Traceback (most recent call last):
File "D:\ExtractMesh\SuGaR\train.py", line 141, in
coarse_mesh_path = extract_mesh_from_coarse_sugar(coarse_mesh_args)[0]
File "D:\ExtractMesh\SuGaR\sugar_extractors\coarse_mesh.py", line 427, in extract_mesh_from_coarse_sugar
decimated_o3d_fg_mesh = o3d_fg_mesh.simplify_quadric_decimation(decimation_target)
MemoryError: bad allocation
——————————————————————————————————————————
I ran the code with 32GB of memory but encountered an issue. May I ask about the memory requirements for running this project？
Looking forward to your reply!

RuntimeError: cannot reshape tensor of 0 elements into shape [0, -1, 1] because the unspecified dimension size -1

Almost finished :) the long refinement, but it unfortunately crashed.
Do you know what might be the error?
Do I need to run now all again? or just from where it crashed is enough?

Thanks in advance.

Loading mesh to bind to:
./output/coarse_mesh\playroom\sugarmesh_coarse\playroom\3Dgs7000_sdfestim02_sdfnorm02_level03_decim1000000.ply...
RPly: Unable to open file
[Open3D WARNING] Read PLY failed: unable to open file: ./output/coarse_mesh\playroom\sugarmesh_coarse\playroom\3Dgs7000_sdfestim02_sdfnorm02_level03_decim1000000.ply
Mesh to bind to loaded.
Binding radiance cloud to surface mesh...
Traceback (most recent call last):
  File "...\SuGaR\train.py", line 162, in <module>
    refined_sugar_path = refined_training(refined_args)
  File "...\SuGaR\sugar_trainers\refine.py", line 393, in refined_training
    sugar = SuGaR(
  File "...\SuGaR\sugar_scene\sugar_model.py", line 323, in __init__
    scales = scales.clamp_min(0.0000001).reshape(len(faces_verts), -1, 1).expand(-1, self.n_gaussians_per_surface_triangle, 2).clone().reshape(-1, 2)
RuntimeError: cannot reshape tensor of 0 elements into shape [0, -1, 1] because the unspecified dimension size -1 can be any value and is ambiguous

Confusion about the learning rate in sugar model

Hi ~ Thanks for your amazing work.
I am confused about the learning rate for gaussian's position in sugar_optimizer.py
why is position_lr_init * spatial_lr_scale ???
From my personal understanding，the "spatial_lr_scale" is the radius, which is acquired from camera positions.
Multiplication will result in a larger scene, a larger radius, a larger spatial_lr_scale and a larger positional learning rate.

And this is the exact opposite of what was described in readme.

Should it be position_lr_init / spatial_lr_scale ?

Looking forward for your reply.
Best regards!

To our great author(s): can we: 1) speed up coarse-training, and no extraction. 2) then dirctly use surface-oriented gauss-points for interactive scenarios (collision detection，morphological changing, ...)

as tilte

I am checking the ‘watertight’ quality, maybe we need another regularization for 'watertight', and tolerance a little bit points-overlay.

when is the code available?

"OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect" ERROR MESSAGE

On the final train.py command, I receive an error and for some reason "sugarfine_C:" is added to path BUT does NOT exist as a file or folder in the directory - Final Output -
....
Traceback (most recent call last):
File "C:\Users\Dirk\SuGaR\train.py", line 160, in
refined_sugar_path = refined_training(refined_args)
File "C:\Users\Dirk\SuGaR\sugar_trainers\refine.py", line 316, in refined_training
os.makedirs(sugar_checkpoint_path, exist_ok=True)
File "C:\Users\Dirk\anaconda3\envs\sugar\lib\os.py", line 215, in makedirs
makedirs(head, exist_ok=exist_ok)
File "C:\Users\Dirk\anaconda3\envs\sugar\lib\os.py", line 215, in makedirs
makedirs(head, exist_ok=exist_ok)
File "C:\Users\Dirk\anaconda3\envs\sugar\lib\os.py", line 215, in makedirs
makedirs(head, exist_ok=exist_ok)
[Previous line repeated 11 more times]
File "C:\Users\Dirk\anaconda3\envs\sugar\lib\os.py", line 225, in makedirs
mkdir(name, mode)
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: 'C:\Users\Dirk\SuGaR\gaussian_splatting\data\head01\sugarfine_C:'

BEST BEST BEST one (NOT one-of) so far (almost all technical directions), but still a few challanges.

prioritized (object static mesh)：

1. robust (the pose accuracy computed by COLMAP requirement is very high)
2. quality (garbage point, connection, normals, ...)
3. speed (2-hours level, goal is 10 minutes)
4. vertex-face number (intermediate-state: 10-million-level, final mesh: 500K ; original mesh is just about 7k)
5. better to have: the mask logic in algorithm native(GS and SuGaR parts) is necessary.

PS. another potential way: add-remove-move vertexes by gradients, and directly use nvdiffrast. (like GS operate gauss-points)

The path error during running the train.py

I've got this error when training the SuGaR model.

---

I solved it by hardcorely chaning this line in refine.py
ply_path = os.path.join(source_path, "points3d.ply") # line 288

I suggest that the path of the code can be clearer.

Little typo in Manual Instructions

The line to install pytorch3d seems to be backwards. It reads:

conda install pytorch3d -c pytorch3d==0.7.4

and should be

conda install pytorch3d==0.7.4 -c pytorch3d

Great work and my utmost respect for this breakthrough!

mesh

Can you generate a mesh with a color texture?

Missing the nerf synthetic dataloader in the sugar train script.

Thanks for the great work.
I found that when I tried to train the sugar model for the NeRF synthetic dataset, the path was wrong.
It seems that the original codebase only supports the COLMAP dataset.
Do you have the dataloader for the nerf synthetic dataset?
Looking for your reply, thanks.

Sharing results from the 3D Gaussian Splatting repository

Hi,

This would be very useful if you could share your result coming from the original 3D Gaussian Splatting repository somewhere (gdrive?), so we can compare ours.

encounter error, something about image loading.

here is my error message, I have no clue what's wrong.

python train.py -s /home/ezxr/data/mapper_sanxdxgcjcs_002617 -c /home/ezxr/data/mapper_sanxdxgcjcs_002617/gaussain_model/ -r density
Will export a UV-textured mesh as an .obj file.
Changing sh_levels to match the loaded model: 4
-----Parsed parameters-----
Source path: /home/ezxr/data/mapper_sanxdxgcjcs_002617

Content: 4
Gaussian Splatting checkpoint path: /home/ezxr/data/mapper_sanxdxgcjcs_002617/gaussain_model/
Content: 7
SUGAR checkpoint path: ./output/coarse/mapper_sanxdxgcjcs_002617/sugarcoarse_3Dgs7000_densityestim02_sdfnorm02/
Iteration to load: 7000
Output directory: ./output/coarse/mapper_sanxdxgcjcs_002617
SDF estimation factor: 0.2
SDF better normal factor: 0.2
Eval split: True

---

Using device: 0
|===========================================================================|

PyTorch CUDA memory summary, device ID 0
CUDA OOMs: 0
===========================================================================
Metric
---------------------------------------------------------------------------
Allocated memory
from large pool
from small pool
---------------------------------------------------------------------------
Active memory
from large pool
from small pool
---------------------------------------------------------------------------
GPU reserved memory
from large pool
from small pool
---------------------------------------------------------------------------
Non-releasable memory
from large pool
from small pool
---------------------------------------------------------------------------
Allocations
from large pool
from small pool
---------------------------------------------------------------------------
Active allocs
from large pool
from small pool
---------------------------------------------------------------------------
GPU reserved segments
from large pool
from small pool
---------------------------------------------------------------------------
Non-releasable allocs
from large pool
from small pool
---------------------------------------------------------------------------
Oversize allocations
---------------------------------------------------------------------------
Oversize GPU segments
===========================================================================

Loading config /home/ezxr/data/mapper_sanxdxgcjcs_002617/gaussain_model/...
Performing train/eval split...
Loading camera 21907/1695894918480.jpeg
Loading camera 21907/1695894919015.jpeg
Loading camera 21907/1695894919681.jpeg
Loading camera 21907/1695894920349.jpeg
Loading camera 21907/1695894921016.jpeg
Loading camera 21907/1695894922349.jpeg
Loading camera 21907/1695894923682.jpeg
Loading camera 21907/1695894924349.jpeg
Loading camera 21907/1695894925016.jpeg
Loading camera 21907/1695894925682.jpeg
Loading camera 21907/1695894926349.jpeg
Loading camera 21907/1695894927016.jpeg
Loading camera 21907/1695894927684.jpeg
Loading camera 21907/1695894928350.jpeg
Loading camera 21907/1695894929016.jpeg
Loading camera 21907/1695894929684.jpeg
Loading camera 21907/1695894930349.jpeg
Loading camera 21907/1695894931016.jpeg
Loading camera 21907/1695894931684.jpeg
Loading camera 21907/1695894932350.jpeg
Loading camera 21907/1695894933016.jpeg
Loading camera 21907/1695894933684.jpeg
Loading camera 21907/1695894934350.jpeg
Loading camera 21907/1695894935017.jpeg
Loading camera 21907/1695894935684.jpeg
Loading camera 21907/1695894937016.jpeg
Loading camera 21907/1695894937684.jpeg
Loading camera 21907/1695894938350.jpeg
Loading camera 21907/1695894939016.jpeg
Loading camera 21907/1695894939684.jpeg
Loading camera 21907/1695894940350.jpeg
Loading camera 21907/1695894941017.jpeg
Loading camera 21907/1695894941687.jpeg
Loading camera 21907/1695894942351.jpeg
Loading camera 21907/1695894943017.jpeg
Loading camera 21907/1695894943684.jpeg
Loading camera 21907/1695894944351.jpeg
Loading camera 21907/1695894945018.jpeg
Loading camera 21907/1695894945686.jpeg
Loading camera 21907/1695894946351.jpeg
Loading camera 21907/1695894947018.jpeg
Loading camera 21907/1695894947686.jpeg
Loading camera 21907/1695894948351.jpeg
Loading camera 21907/1695894949018.jpeg
Loading camera 21907/1695894949686.jpeg
Loading camera 21907/1695894950350.jpeg
Loading camera 21907/1695894951018.jpeg
Loading camera 21907/1695894951686.jpeg
Loading camera 21907/1695894952351.jpeg
Loading camera 21907/1695894953019.jpeg
Loading camera 21907/1695894953686.jpeg
Loading camera 21907/1695894954351.jpeg
Loading camera 21907/1695894955018.jpeg
Loading camera 21907/1695894955685.jpeg
Loading camera 21907/1695894956352.jpeg
Loading camera 21907/1695894957019.jpeg
Loading camera 21907/1695894957687.jpeg
Loading camera 21907/1695894958352.jpeg
Loading camera 21907/1695894959019.jpeg
Loading camera 21907/1695894959687.jpeg
Loading camera 21907/1695894960352.jpeg
Loading camera 21907/1695894961019.jpeg
Loading camera 21907/1695894961687.jpeg
Loading camera 21907/1695894962352.jpeg
Loading camera 21907/1695894963019.jpeg
Loading camera 21907/1695894963714.jpeg
Loading camera 21907/1695894964352.jpeg
Loading camera 21907/1695894965019.jpeg
Loading camera 21907/1695894965687.jpeg
Loading camera 21907/1695894966352.jpeg
Loading camera 21907/1695894967019.jpeg
Loading camera 21907/1695894967687.jpeg
Loading camera 21907/1695894968352.jpeg
Loading camera 21907/1695894969019.jpeg
Loading camera 21907/1695894969687.jpeg
Loading camera 21907/1695894970353.jpeg
Loading camera 21907/1695894971020.jpeg
Loading camera 21907/1695894971687.jpeg
Loading camera 21907/1695894972353.jpeg
Loading camera 21907/1695894973020.jpeg
Loading camera 21907/1695894973687.jpeg
Loading camera 21907/1695894974352.jpeg
Loading camera 21907/1695894975020.jpeg
Loading camera 21907/1695894975687.jpeg
Loading camera 21907/1695894976353.jpeg
Loading camera 21907/1695894977022.jpeg
Loading camera 21907/1695894977687.jpeg
Loading camera 21907/1695894978353.jpeg
Loading camera 21907/1695894979020.jpeg
Loading camera 21907/1695894979687.jpeg
Loading camera 21907/1695894980354.jpeg
Loading camera 21907/1695894981020.jpeg
Loading camera 21907/1695894981690.jpeg
Loading camera 21907/1695894982353.jpeg
Loading camera 21907/1695894983020.jpeg
Loading camera 21907/1695894983688.jpeg
Loading camera 21907/1695894984354.jpeg
Loading camera 21907/1695894985020.jpeg
Loading camera 21907/1695894985689.jpeg
Loading camera 21907/1695894986357.jpeg
Loading camera 21907/1695894987020.jpeg
Loading camera 21907/1695894987689.jpeg
Loading camera 21907/1695894988353.jpeg
Loading camera 21907/1695894989020.jpeg
Loading camera 21907/1695894989687.jpeg
Loading camera 21907/1695894990353.jpeg
Loading camera 21907/1695894991021.jpeg
93 training images detected.
The model has been trained for 7000 steps.

Camera resolution scaled to 360 x 640
Initializing model from trained 3DGS...
Point cloud generated. Number of points: 252966
Use min to initialize scales.
Initialized radiuses for 3D Gauss Rasterizer
Initializing 3D gaussians from 3D gaussians...

SuGaR model has been initialized.
SuGaR()
Number of parameters: 14924994
Checkpoints will be saved in ./output/coarse/mapper_sanxdxgcjcs_002617/sugarcoarse_3Dgs7000_densityestim02_sdfnorm02/

Model parameters:
_points
torch.Size([252966, 3])
True
all_densities
torch.Size([252966, 1])
True
_scales
torch.Size([252966, 3])
True
_quaternions
torch.Size([252966, 4])
True
_sh_coordinates_dc
torch.Size([252966, 1, 3])
True
_sh_coordinates_rest
torch.Size([252966, 15, 3])
True
Using camera spatial extent as spatial_lr_scale: 7.771456003189088
Optimizer initialized.
Optimization parameters:
OptimizationParams(
iterations=15000,
position_lr_init=0.00016,
position_lr_final=1.6e-06,
position_lr_delay_mult=0.01,
position_lr_max_steps=30000,
feature_lr=0.0025,
opacity_lr=0.05,
scaling_lr=0.005,
rotation_lr=0.001,
)
Optimizable parameters:
points 0.001243432960510254
sh_coordinates_dc 0.0025
sh_coordinates_rest 0.000125
all_densities 0.05
scales 0.005
quaternions 0.001
Densifier initialized.
Using loss function: l1+dssim
Starting regularization...

---

Iteration: 7000
loss: 0.030430 [ 7000/15000] computed in 0.0077037890752156574 minutes.
------Stats-----
---Min, Max, Mean, Std
Points: -36.427547454833984 25.382139205932617 0.1579069197177887 4.527556419372559
Scaling factors: 1.8439502014189202e-07 0.9512976408004761 0.032854754477739334 0.05896857753396034
Quaternions: -0.9968045353889465 0.9999953508377075 0.22015613317489624 0.4489225745201111
Sh coordinates dc: -2.2877933979034424 4.295931339263916 -0.06274425983428955 1.2825195789337158
Sh coordinates rest: -0.7043451070785522 0.6325781345367432 0.0003956133150495589 0.049302585422992706
Opacities: 0.0008357342449016869 1.0 0.23739202320575714 0.2816372513771057

---INFO---
Starting entropy regularization.

---INFO---
Resetting neighbors...
Traceback (most recent call last):
File "/home/mm/ARWorkspace/SuGaR/train.py", line 120, in
coarse_sugar_path = coarse_training_with_density_regularization(coarse_args)
File "/home/mm/ARWorkspace/SuGaR/sugar_trainers/coarse_density.py", line 526, in coarse_training_with_density_regularization
gt_rgb = gt_image.view(-1, sugar.image_height, sugar.image_width, 3)
RuntimeError: shape '[-1, 360, 640, 3]' is invalid for input of size 689280

Too Slow training

Hi, Thanks for your great work and the open source code. I encountered too slow training when doing training with my RTX 3090 machine. And it will take 5~6 minutes to do 50 iterations (8000 in total). And the whole training would take like over 10 hours. That's way longer than what it is in the paper. Am I miss doing something?

question to to Eq.4

"diagonal matrix with only one nonzero value on the diagonal (the other two being much smaller)"
@ #2

‘flat’: only-one nonzero OR “at-most-two nonzero” ?

at-most-two nonzero includes:
2-nonzero: flat / plane
1-nonzero: thin / line

for example, a single Gaussian point, the scaling is x_y_z: [0.1, 0.2, 0.3]
In Eq.4, does the s_g mean 0.1, and n_g mean 0 : x-axis?

Thanks

unexpected results

Hi, i'm trying to reproduce your excellent mesh reconstruction. But got an unexpected result instead.
I used the data with colmap provided by the original 3D Gaussian Splatting code. And obtain the truck as
(open in Blender)

(open in meshlab)

My commands:
python gaussian_splatting/train.py -s ./truck/ --iterations 7000 -m output/truck
python train.py -s truck/ -c output/truck/ -r density

I had try "-r sdf" and the result was about the same.
The defect is also exists in other reconstructions like playroom.

Could you give me some suggestions?
Thanks!

great !!!

Question about entropy loss

Hi, it's really exciting to see this awesome work!

I have a question about the entropy loss in the paper. How do you calculate the entropy loss, as there are no groud truth density provided?

Question about the implementation details

Hi

Thanks for your nice work. I have some implementation details about how to get the nearest g* in equation (3). Hope this can get the details from you.

You mentioned this in Sec.5.1

To compute the density values of points
from a gaussian g, we sum only the gaussian functions from
the 16 nearest gaussians of g and update the list of nearest
neighbors every 500 iterations.

For the regularization from 9000-15000 steps, you will sample different p from the gaussian distribuion in equation (8).
Therefore, I wonder how you can keep the nearest list of g unchanged for each p at different iteration(update every 500 iterations)? Based on my understanding, the p sampled from each splats varied at different iterations.

Thanks in advance
Yiqun

A little bug fixed: tensor as index should be long

point_idx_per_pixel = torch.zeros(texture_size, texture_size, device=rc.device).int()
-->
point_idx_per_pixel = torch.zeros(texture_size, texture_size, device=rc.device).long()

torch.tensor(np.array(surface_mesh_to_bind.triangles)).to(nerfmodel.device),
-->
torch.tensor(np.array(surface_mesh_to_bind.triangles), dtype=torch.long).to(nerfmodel.device),

--export_uv_textured_mesh / -t ERROR

Success on install of SuGaR

Success on convertr.py and train.py in quick start.

Problem: cannot convert gaussian splat trained into the textured mesh (--export_uv_textured_mesh / -t )-

I first tried this command to convert to obj/texture -
(sugar) C:\Users\Dirk\SuGaR>python train.py -s data/head01 -c output\ -r density
[FAILED]

Then I tried -
python -Bu train_coarse_density.py -s data/head01 -c output
[FAILED]

Both of the attempts gave ERROR -
train_coarse_density.py: error: unrecognized arguments: -t output\

a snippet of cmd prompt results, I also show 3 different iterations of prompt structures I tried seperated by double spaces , all wrong -

Loading mesh to bind to:
./output/coarse_mesh\head01\sugarmesh_coarse\head01\3Dgs7000_densityestim02_sdfnorm02_level03_decim1000000.ply...
RPly: Unable to open file
[Open3D WARNING] Read PLY failed: unable to open file: ./output/coarse_mesh\head01\sugarmesh_coarse\head01\3Dgs7000_densityestim02_sdfnorm02_level03_decim1000000.ply
Mesh to bind to loaded.
Binding radiance cloud to surface mesh...
Traceback (most recent call last):
File "C:\Users\Dirk\SuGaR\train.py", line 162, in
refined_sugar_path = refined_training(refined_args)
File "C:\Users\Dirk\SuGaR\sugar_trainers\refine.py", line 393, in refined_training
sugar = SuGaR(
File "C:\Users\Dirk\SuGaR\sugar_scene\sugar_model.py", line 323, in init
scales = scales.clamp_min(0.0000001).reshape(len(faces_verts), -1, 1).expand(-1, self.n_gaussians_per_surface_triangle, 2).clone().reshape(-1, 2)
RuntimeError: cannot reshape tensor of 0 elements into shape [0, -1, 1] because the unspecified dimension size -1 can be any value and is ambiguous

(sugar) C:\Users\Dirk\SuGaR>python -Bu train_coarse_density.py -s data/head01 -c output\ -t output
usage: train_coarse_density.py [-h] [-c CHECKPOINT_PATH] [-s SCENE_PATH] [-o OUTPUT_DIR] [-i ITERATION_TO_LOAD]
[--eval EVAL] [-e ESTIMATION_FACTOR] [-n NORMAL_FACTOR] [--gpu GPU]
train_coarse_density.py: error: unrecognized arguments: -t output\

(sugar) C:\Users\Dirk\SuGaR>python -Bu train_coarse_density.py -s data/head01 -t output
usage: train_coarse_density.py [-h] [-c CHECKPOINT_PATH] [-s SCENE_PATH] [-o OUTPUT_DIR] [-i ITERATION_TO_LOAD]
[--eval EVAL] [-e ESTIMATION_FACTOR] [-n NORMAL_FACTOR] [--gpu GPU]
train_coarse_density.py: error: unrecognized arguments: -t output\

(sugar) C:\Users\Dirk\SuGaR>python -Bu train_coarse_density.py -s data/head01 -t output\ -r density
usage: train_coarse_density.py [-h] [-c CHECKPOINT_PATH] [-s SCENE_PATH] [-o OUTPUT_DIR] [-i ITERATION_TO_LOAD]
[--eval EVAL] [-e ESTIMATION_FACTOR] [-n NORMAL_FACTOR] [--gpu GPU]
train_coarse_density.py: error: unrecognized arguments: -t output\ -r density

What is the difference between $\hat{f}$ and $\bar{f}$?

Hi @Anttwo,

Congratulations on your preprint! Results look great. I have a couple of questions:

1. I have a confusion about $\hat{f}$ and $\bar{f}$. What is the difference between them? Is $\hat{f}$ an approximation/efficient version of $\bar{f}$?
2. How many points do you sample per Gaussian? To be more specific, I am asking the number of points sampled to obtain set $\mathcal{P}$.
3. How do you compute the depth given multiple Gaussians with different $\alpha_g$?

can we use this type of regularization to flatten gauss-point ball?

import torch
x = torch.tensor([1.0], requires_grad=True) #internal auxiliary parameter for X
y = torch.tensor([1.1], requires_grad=True) #internal auxiliary parameter for Y
optimizer = torch.optim.SGD([x,y], lr=1.0)
for i in range(100):
X = torch.sigmoid(x) #compent_x @scale
Y = torch.sigmoid(y) #compent_y @scale
reg1 = torch.nn.functional.mse_loss(X+Y, torch.tensor([1.0])) #x+y=1
reg2 = 1./(XX + YY) #one >> other(s)
mse0 = 1.0 #GS optimization (dummy)
loss = reg1*10. + reg2 + mse0
optimizer.zero_grad()
loss.backward()
optimizer.step()
print("i=%02d X=%.2f Y=%.2f reg1=%.4f reg2=%.4f loss=%.4f"%(i, X.tolist()[0], Y.tolist()[0], reg1.item(), reg2.item(), loss.item()))

bad result on the bicycle scene?

Thanks for your great work!
I got good result in garden scene, but bad result on bicycle. I tried density and sdf regularization_type, it doesn't work.
What is the cause of this?

Hope to get your reply!

What is the better training policy for SuGaR?

1. Train GS first, then joint optimization. (Although the position of GS points is not very good, it is generally oriented towards shape and surface.)
2. Train GS and SuGaR regurlization item, joint optimization from begining. (I can not estimate the SuGaR item computing cost before code ready.) This way maybe faster maybe slower, maybe quality bettwer maybe worse。
3. A better mixed scheduling policy.

There is a lot of noise on the surface of the mesh

I used the lego scene reproduction experiment in nerf_synthetic and successfully extracted the mesh, but the quality seemed unable to reach the quality in the project page.

cmd ： python train.py -s data/nerf_synthetic/lego/ -c output/lego/ -r sdf
Initialize with random point cloud

how can I use the coarse 15000.pt to continue train.py?

My gpu was out of memory when the train.py is running(temporarily), but I got the "coarse/data/15000.pt",
Can I ignore # ----- Optimize coarse SuGaR ----- #step and run # ----- Extract mesh from coarse SuGaR -----# step directly by setting any parameters?

Question about the joint refinement of mesh and gaussians

Hi, thank you so much for the great work!

I have a few questions regarding optimization process after binding new Gaussians to the mesh.

1. In this refinement stage, how do you choose the initial values of the parameters (SH, opacity, scale, rotation) of the new Gaussians? Do the new Gaussians have connections with the Gaussians from the last optimization stage?

2. If I undertand it correctly, during the optimization, the Gaussians will always be aligned to the mesh triangle and can be moved freely on the surface. Then how about the mesh, will the position of the mesh triangle also be moved or refined? Or does it serve as a process for smoothing the surface, but the geometry of the surface won't be changed too much.

Thanks in advance!