quadjr/aframe-gaussian-splatting

A-Frame component implementation of the 3D Gaussian splat viewer

aframe-gaussian-splatting's Issues

Rendering on Android is incorrect / low resolution

Looks like when rendering on some Android devices the output does resemble the splats but looks very low resolution; not sure if incorrect sorting or if something with the data goes wrong. I tested this also with the live page from this repo.

Tested on Xiaomi 11T. Works fine on other Android devices like Quest 3

Some images of how it looks (after all splats have been loaded):

Image 1	Image 2

There is no way to Preload a splat file.

Normally in aframe , for example, if we have a video file we can preload that file in <a-assets></a-assets>
like

<a-assets>
     <video src="<video source url>" id="myVideoUrl" />
</a-assets>

<a-video src="#myVideoUrl"></a-video>

Here in the case of video, the actual file is preloaded but in the case of splat file with the current stage of development, there is no way to preload splat assets.

Improve render order / sorting for a-frame scenes with multiple splats

As an a-frame developer adding multiple gaussian splats to a scene, I would like splats to naturally occlude each other as though they were placed in real space together.

Issue: By default for transparent objects, A-Frame will use the DOM position to determine render order. This results in scenes where certain camera angles will show an object in front of another even though that object is further away and should be occluded. [picture1]

Workaround: As a scene creator I can carefully arrange entity dom order and entity position / camera position to arrange a scene to look "correct" from specific angles. However this is not a solution for immersive / vr as the user can easily change orientation to see views that do not look correct.

Instead: as a user when I add multiple splats to a scene I should see the rendering of the splats sorted to look natural as though they were part of the same scene

I attempted to solve this problem with using A-Frame sorting for transparent objects. It solves the problem in some contexts but also results in undesired behavior such as entities "popping" in or out of visibility quickly. [video1]

Some other possible solutions:

• Combine splat data together from all splats in the scene for the purpose of sorting for rendering. A-Frame has the concept of a "system" to link multiple components together. This can be a mechanism to allow fetching the splats together and batching the sorting to happen just 1 time per frame instead of sorting multiple times for separate entities which may also result in performance improvement.
• ?

picture1:

video1:
https://github.com/quadjr/aframe-gaussian-splatting/assets/470477/66474fe6-1911-4012-a9de-f25dc9c00a8b

Sorting looks incorrect for vertical axis

Seems while sorting is fine in the horizontal axis, it's incorrect in the vertical axis:
https://github.com/quadjr/aframe-gaussian-splatting/assets/2693840/d0330b01-9728-49dc-938b-5955be10b29e
(same with the train scene)

2 fps on Quest 2

You are the best for creating this, I'm sure you'll will find a solution for optimisation. Good luck.

Performance ideas / benchmarking

Some ideas in regards to performance. Ultimately it would be nice to get (a subset of) this to work on a Quest 2 / 3; currently that's running at 5-10 fps and very choppy. So are the other three.js implementations though!

• using compressed data at runtime. Seems you started on this already! There are some ideas regarding compression formats here: https://aras-p.info/blog/2023/09/27/Making-Gaussian-Splats-more-smaller/
• using better interleaved data so data fetching on the GPU is more localized (same link above has some info)
• using alpha hashing instead of transparency, and then rendering back-to-front instead to get some early Z cutoff
• some kind of LOD system - not sure if splats could be sorted by "importance" (e.g. less transparent ones are more important?) at runtime, or if the calculations would need to be done with less splats in the first place.

Regarding loading behaviour, I've dabbled a bit with creating splats already while loading, will see if I can make a PR for parts of that.

And it would be interesting to load compressed data, again Aras (link above) has some ideas around that and tooling to generate byte buffers that are already optimized (10-20x size reduction).

Wanna join us on Discord?

Great work. We've been having a lot of fun with your component. There's parallel work and great ideas for features and improvements that might make sense to submit here. Would you like to join the chat on the WebXR Discord #a-frame channel?

If camera component is added to entity in scene, splat will not render

Here's a simplified example of code that will prevent splat from rendering:

    <a-scene>
      <a-entity gaussian_splatting="src: https://huggingface.co/cakewalk/splat-data/resolve/main/train.splat;"></a-entity>
      <!-- the following line makes the splat not render -->
      <a-entity id="player" camera position="0 1.6 0" look-controls></a-entity>
    </a-scene>

Error reproduction: https://glitch.com/edit/#!/phantom-copper-sphere?path=index.html%3A15%3A0
(remove line 13 to see it work correctly)

console error:

index.js:16 Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'el')
    at n.init (index.js:16:40)
    at n.initComponent (component.js:318:10)
    at n.updateProperties (component.js:300:12)
    at e.exports.Component (component.js:76:8)
    at new n (component.js:660:15)
    at d.initComponent (a-entity.js:319:17)
    at d.updateComponent (a-entity.js:471:10)
    at d.updateComponents (a-entity.js:442:12)
    at a-entity.js:243:12
    at a-node.js:150:17

Documentation of workflow for custom scans?

As an A-Frame developer creating a custom scene using gaussian splatting I would like to understand how to create my own files.

It would be helpful to add documentation on a "known good" process to scan custom objects.

So far my best workflow is:

• Use https://lumalabs.ai/ to process a scan, I have found video to work very well
• Go to download dialog and choose Gaussian > Splat (which will actually download a .zip file with a .ply file inside).
• use this site to convert the ply to splat in the browser: https://splat-converter.glitch.me/ by @akbartus (repo https://github.com/akbartus/Gaussian-Splatting-WebViewers/tree/main/splat_converter)
• use resultant splat in my a-frame scene

WebXR Navigation, Flight

Controller Mesh Preview is not necessary
Hold any button on Right Controller to fly in the direction of the Right Controller
Hold any button on Left Controller to fly in the direction of the Left Controller
That's all, really simple and a really essential feature!
no rush.
Thanks!

Collaboration...

allow download of cropped splat

As a creative developer creating a 3d scene composed of one or more gaussian splats, I'd like to use a dynamic cutout to place target object and crop unwanted splats, and then to the export a file that only contains the target splats thus discarding the others and creating a much smaller file for use in a scene.

Mentioned in this ticket under future ideas:

allow exporting a lossless PLY again that has "baked" root transformation and cutouts and thus contains less data - suddenly this becomes a nice web editor for PLYs :)

Support cutout box shape in a-frame?

Hi there, I noticed the addition of cutout support for three.js but I don't see a mechanism to define the cutout area via A-Frame.

https://github.com/quadjr/aframe-gaussian-splatting/blob/main/index.js#L431

Here it refers to this.cutout but I don't see that defined anywhere.

Perhaps we could add an optional property to the component of type selector to choose an a-frame entity box primitive (that should already be in the scene) and then set this.cutout to be the three.js object3d of that primitive?

Freezes on scenes larger than the train scene

I tried the code with this model: https://lumalabs.ai/capture/c6f09528-f758-45c7-a92a-4af38067e266 (direct download link: https://drive.google.com/file/d/1p_eveZ66JM7TCKF2swDv1zctzIRo05uK/view?usp=sharing), which works fine in the original antimatter viewer when dropped: https://antimatter15.com/splat/
This viewer also works fine with the file: https://github.com/mkkellogg/GaussianSplats3D

However, when running it with the a-frame viewer component code, everything freezes my machine (M2 Mac) to a halt, and I have to kill the browser process. Would be curious if you get different results and/or have an idea what the reason may be.

how to load ply splats?

i tried opening a ply gaussian splat file exported from luma ai (lumalabs.ai)
and i'm getting this:

Uncaught (in promise) RangeError: byte length of Float32Array should be a multiple of 4
at new Float32Array ()

Thanks, You're the best <3