mkkellogg/gaussiansplats3d

Three.js-based implementation of 3D Gaussian splatting

gaussiansplats3d's Issues

data expired?

https://github.com/mkkellogg/GaussianSplats3D/blob/main/projects.markkellogg.org/downloads/gaussian_splat_data.zip

Is it possible to invert the vertical rendering of a SplatMesh object?

When using an external scene, the SplatMesh object does not react in any way to changes in the camera's Up vector. All examples in the description use the negative direction of the camera Y axis, but in ThreeJS the positive Y direction is the default. Can you tell me where to change your code in order to either implement a dependence on the Up vector of the camera or at least change the direction of the SplatMesh rendering?

Support more splat compressions levels

Some splat imported from other compressors have different compression levels.
namely splats imported from playcanvas compressor have 190 as compression level and splats from polycam website have 97 as level and in SplatBuffer.js at line 58 it throws an error as it's not able to load these files.
is there any way to fix this behaviour?

Set internal render resolution?

Hi, im wondering if theres a way in the code to set the internal render resolution, so maintain the same size viewer but for example set the model itself to be at 0.5* the resolution?

Discussion: remove transparent splats during data conversion?

I've just written my own renderer:
https://github.com/fynv/web_gaussian_splats
, which is heavily based on the great work here. Thanks very mcuh!

One thing I noticed is that if I don't remove the splats with alpha == 0,
there can be some obvious performance defficiency (fragment shader bound).
The indication is that there are a signicant number of splats with alpha ==0.

The culling is currently done when building the "splat-tree" structure, where "splatAlphaRemovalThreshold" is defaulted to 1.
It seems to me that for alpha == 0, it doesn't even make sense to store these splats in the .splat data in the first place.
Can we just drop these splats during data conversion?

Depth sorting for foreign objects

Is it possible to make depth sorting apply to added 3b objects? For example debug meshes in demo are rendered even when occluded by terrain

Typescript support

Hey,

I'd like to hear your stance on this topic.

Personally I think it's a great developer tool and a great aid when exposing the API to the end user.

I'd happily convert the existing code if you'd like me to.

how to select coordinate points in the Splat scene

There is an issue that has been bothering me, which is how to select coordinate points in the Splat scene through mouse clicks in order to conveniently add regular meshes to the scene. However, when I tried to implement it using the raycaster.intersectObjects() method, it didn’t work.

Not rendering local host

After:

npm run demo

I get:

Server running at 127.0.0.1:8080

---

HTTP(500)) Request for ./build/demo/ -> Server error.
HTTP(404) Request for ./build/demo/favicon.ico -> File not found.

When I navigate to local host URL it says this:

Sorry, check with the site admin for error: EISDIR ..

When I used SERVEZ running locally I get the index page to load but the links to the three example pages don't load splats.

I have demo data installed here:

GaussianSplats3D-main\build\demo\assets\data

\garden
\stump
\truck

All three directories have associated splat files.

Any ideas what I might be doing wrong?

how could i do to establish an online demo?

Thx for your great work !
I have run successfully locally,
How could i do to run it online?

thx for your reply!

The custom camera's "lookat" parameter is not working.

hot reload with bun

Hi is it possible to get hot-reload working with it?

i am using bun run build and bun run demo

but a hot reload will be awesome to have

https://bun.sh/guides/http/hot

Load another Splat resource

I am loading another Splat in the garden scene, which may cause occlusion issues. When I move the camera's perspective, the table will disappear, and I cannot find the reason for it

Feature Request: FPS widget in the corner with option to hide

Something like this, i think threejs has this already. would be very useful.

Rendering on Smartphone

Super nice work, it renders super smooth on my Macbook pro :)

I "optimized "the ply to splat with your function but it still doens't render on my android phone (not super fast but not slow Pixel 7)

Any ideas of optimization? I tested the 3 examples too, it doesn"t work.

Thanks!

3D Doesn't Appear

Hello. I was looking to work on Gaussian Splat with Three.js and found this GitHub repository.
However, I immediately ran into a major obstacle.
Even after installing and uploading to the server, and testing with the codes provided in the sample, I can't see the ksplat on my server. The splat files we worked on also don't upload.
In the sample page console, there are a few more Views.js files that appear, but
Views.js doesn't show up in our console.
What could be the problem?

` import * as THREE from 'three';
import * as GaussianSplats3D from 'gaussian-splats-3d';

const viewer = new GaussianSplats3D.Viewer({
    'cameraUp': [0, -1, -0.54],
    'initialCameraPosition': [-3.15634, -0.16946, -0.51552],
    'initialCameraLookAt': [1.52976, 2.27776, 1.65898]
});
viewer.init();
let path = './shoose.splat';
viewer.loadFile(path, {
    'halfPrecisionCovariancesOnGPU': true
})
.then(() => {
    viewer.start();
});`

Splats disappearing depending on tilt

Hi @mkkellogg !

I'm experiecing some strange behaviour and i was wondering if you could help me/had an idea of what could be happening.

I have a ksplat that i'm trying to playback but depending on the tilt of the camera a lot of the splats disappear.
This doesn't happen when using the SIBR viewer (or antimatter15 implementation for what i can tell, but i might be wrong, as i just used the online viewer and is not too user friendly).

For what is worth i think it's to do with the sorting: i removed the toHalfFloat for the covariance and i could see all the points without the splats, and they were still disappering. Then again, i'm not sure why just sorting the splats would make them disappear, so maybe it's something else.

I attacched a test i've done:
converted_file.zip
If you try with:
'cameraUp': [0, 0, -1],
'initialCameraPosition': [9993.60576, -277.27384, -435.28904],
'initialCameraLookAt': [9293.52797, 251.08879, 28.23262],

and slightly tilt up/down you should see them disappearing

Note on the file: when i was playing around with the bucketSize and having huge files back, i did something to cull out the model given a point and a radius.
What i've attacched is just a culled version fo the entire model, i was trying to check if there was something "in front" that would occlude the rest, i can assure you this happens on the full model as well, this is just simplier to attach 😄

Scale parameter

Hi @mkkellogg !

I was wondering what was your opinion on adding another parameter that can be passed to the viewer and that would tide together MaximumDistanceToSort/MaximumDistanceToRender/SplatBufferBucketBlockSize and the Camera.far?

Is there any drawback?
Did i miss some other variable?

Convert .ply to .splat

Dear author,
I'm a new user for java. Could you please tell me where to put the code you provide to convert .ply file to .splat file? Thank you.
Best regards,
hhc

error build

npm run build

[email protected] build
npm run build-library && npm run build-demo

[email protected] build-library
npx rollup -c && mkdir -p ./build/demo/lib && cp ./build/gaussian-splat-3d* ./build/demo/lib/

./src/index.js → ./build/gaussian-splat-3d.umd.cjs...
(!) Broken sourcemap
https://rollupjs.org/troubleshooting/#warning-sourcemap-is-likely-to-be-incorrect
Plugins that transform code (such as "base64") should generate accompanying sourcemaps.
created ./build/gaussian-splat-3d.umd.cjs in 131ms

./src/index.js → ./build/gaussian-splat-3d.module.js...
(!) Broken sourcemap
https://rollupjs.org/troubleshooting/#warning-sourcemap-is-likely-to-be-incorrect
Plugins that transform code (such as "base64") should generate accompanying sourcemaps.
created ./build/gaussian-splat-3d.module.js in 70ms
命令语法不正确。

docker port trouble

Thanks for the great development.
When I create an environment with docker container in wsl on windows, if the port number mapped in docker-compose.yml and the port number described in server.js are the same, the server cannot start and does not output any errors. I imagine this is related to something in this library. Do you have any idea what it could be?

For example, in the following example, node server.js will not start, and i must either change the mapping to 8080:8081 or devise a way to change the port specification in server.js to 8082.

version: '3' services: app: container_name: app build: context: . dockerfile: Dockerfile restart: always tty: true volumes: - type: bind source: . target: /app ports: - 8080:8080

Can `Cesium` support displaying `.ksplat` model files?

The compression effect of .ksplat model on .ply is great! I wonder if it is possible to display .ksplat models in Cesium?

For example, displaying a .ksplat model of a building on Cesium using a Cesium plugin, and ensuring compatibility with Cesium's browsing features like zooming, panning, and rotating...

Local demo fails because of missing splats

When I want to test the demos on my local clone, after having run npm run build there is no folder ./build/demo/assets/data/stump/[*.splat]. Therefore running npm run demo will eventually not open any demo samples.

Not sure if I am doing something wrong, but if not, would there be any chance to include the demo splats in the repo?

WebXR support for Quest 3

Hey great work here so far @mkkellogg!

I'd love to help out where we can on a WebXR viewer for Splats.
We have our own internal renderer right now running on WebXR but it's closed source. Ideally we could support a open source renderer in a popular framework like Three.js to make them more accessible.

We have optimized splats to make them really small while retaining most of the quality. We have the splat here in the repo running at 90FPS on Quest 3 in our WebXR renderer. Hope this can be useful for testing out in the headset since most of the data out there is too large to be useful.
https://github.com/gmix-tech/small_splats

Let me know how I can help out.
Feel free to ping me at [email protected]

Feature: Export `.ksplat` to `.ply`

If a .ply file is converted to .ksplat, can the .ksplat file be converted back to .ply without the original .ply file?

For the Docu: Vite and CORS

In case somebody uses Vite: Vite seems to require some special CORS configuration to be able to read SharedBufferArray. A solution would be this:
chaosprint/vite-plugin-cross-origin-isolation#3
or this vite plugin:
https://github.com/chaosprint/vite-plugin-cross-origin-isolation

The results are different when rotated by dynamicScene and when rotated by addSplatScene.

I tried to rotate 180° along the Z axis in the model direction.
I tried to rotate using addSplatScene with params.

<!DOCTYPE html>
<html lang="en">

<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <meta http-equiv="x-ua-compatible" content="ie=edge">
  <title>3D Gaussian Splat Demo - Garden</title>
  <script type="text/javascript" src="js/util.js"></script>
  <script type="importmap">
    {
        "imports": {
            "three": "./lib/three.module.js",
            "@mkkellogg/gaussian-splats-3d": "./lib/gaussian-splats-3d.module.js"
        }
    }
  </script>
  <style>

    body {
      background-color: #000000;
      height: 100vh;
      margin: 0px;
    }

  </style>

</head>

<body>
  <script type="module">
    import * as GaussianSplats3D from '@mkkellogg/gaussian-splats-3d';
    import * as THREE from 'three';

    const quaternion = new THREE.Quaternion();
    const euler = new THREE.Euler( 0.0, 0.0, Math.PI);
    quaternion.setFromEuler(euler);

    const viewer = new GaussianSplats3D.Viewer({
      "dynamicScene": false,
    });
    let path = 'assets/data/garden/garden';
    path += isMobile() ? '.ksplat' : '_high.ksplat';
    viewer.addSplatScene(path,{
      "rotation": quaternion.toArray()
    })
    .then(() => {
        viewer.start();
    });
  </script>
</body>

</html>

I got the result as shown in the following image.

Next, I tried to rotate using dynamicScene.

<!DOCTYPE html>
<html lang="en">

<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <meta http-equiv="x-ua-compatible" content="ie=edge">
  <title>3D Gaussian Splat Demo - Garden</title>
  <script type="text/javascript" src="js/util.js"></script>
  <script type="importmap">
    {
        "imports": {
            "three": "./lib/three.module.js",
            "@mkkellogg/gaussian-splats-3d": "./lib/gaussian-splats-3d.module.js"
        }
    }
  </script>
  <style>

    body {
      background-color: #000000;
      height: 100vh;
      margin: 0px;
    }

  </style>

</head>

<body>
  <script type="module">
    import * as GaussianSplats3D from '@mkkellogg/gaussian-splats-3d';
    import * as THREE from 'three';

    const quaternion = new THREE.Quaternion();
    const euler = new THREE.Euler( 0.0, 0.0, Math.PI);
    quaternion.setFromEuler(euler);

    const viewer = new GaussianSplats3D.Viewer({
      "dynamicScene": true,
    });
    let path = 'assets/data/garden/garden';
    path += isMobile() ? '.ksplat' : '_high.ksplat';
    viewer.addSplatScene(path)
    .then(() => {
        const splatScene = viewer.getSplatScene(0);
        splatScene.quaternion.copy(quaternion);
        viewer.start();
    });
  </script>
</body>

</html>

I got the result as shown in the following image.

I wanted to reproduce the result of dynamicScene with addSplatScene.
How can I reproduce this using addSplatScene? Also, is this behavior the correct behavior for the inconsistency?

When zooming out the scene using the mouse, some objects start to disappear at a certain distance.

When using the mouse wheel to zoom in on the scene, some objects disappear partially. I have tried setting the camera's far clipping plane to 2000, but it doesn't seem to have any effect.

Error: export 'default' (imported as 'SorterWasm') was not found in './sorter.wasm'

Hi. Can you tell me what sorter.wasm file is for and how to compile it? I'm trying to build this package in my Angular project. When trying to create a build, the following errors appear:

./src/app/services/engine/GaussianSplatsViewer/worker/SortWorker.js:96:38-48 - Error: export 'default' (imported as 'SorterWasm') was not found in './sorter.wasm' (possible exports: __wasm_call_ctors, emscripten_tls_init, sortIndexes)
./src/app/services/engine/GaussianSplatsViewer/worker/sorter.wasm - Error: Module not found: Error: Can't resolve 'env' in 'E:\Projects\artwork-threed-visualizer\VoyagerFrontend\src\app\services\engine\GaussianSplatsViewer\worker'

I see the compile_wasm.sh script, I ran it using PowerShell but no new file was generated. I also don’t see anything in the package.json file that relates to wasm. Can you tell me what these files are needed for and how to configure the build correctly?
Or is sorter.wasm just a binary file that needs to be imported into a JS file as a string?
EDIT:
It doesn't matter anymore. I used fetch to download this file. And to fix the error with self.crossOriginIsolated, I set some headers in the angular.json file. Maybe this will help someone in the future.

Viewer.js
fetch('assets/libs/sorter.wasm').then((response) => response.arrayBuffer())
                .then((sortWorkerBuffer) => {
                    const base64String = this.arrayBufferToBase64(sortWorkerBuffer);
...
});

angular.json
"serve": {
          "builder": "@angular-builders/custom-webpack:dev-server",
          "options": {
            "browserTarget": "artwork-threed-visualizer:build",
            "headers": {
              "Cross-Origin-Opener-Policy": "same-origin",
              "Cross-Origin-Embedder-Policy": "credentialless"
            }
          },

Sorry, check with the site admin for error: EISDIR ..

I run successly in my local computer,but it shows the bug. anyone can help me？

Camera orientation settings not clear

Nicely done ...

Unfortunately, at the moment with the View Your Own Scene on the demo page, it's very unintuitive as to how to set the Camera orientation parameters to get an uploaded splat view to change with the controls as expected - I tried half a dozen times, and always failed. Would it be possible to make this clearer, or easier to use?

Feature: Support for joystick (PS5, Xbox)

very useful to map this to unity, unreal type controls with joystick.

Camera up vs Orientation behavior

Problem description:

Setting camera UP vector on file loading works great for orienting my model. But I would love to use the orientation feature instead so the model can exist in default world space. However, it does not appear to work how I'd expect. I would assume it would yield the same visual effect as changing the camera up i.e. the camera orbits the camera on the correct plane.

Can you confirm the expected behavior/functionality and usage?

Relevant code:

const up = [0.1294521689414978, -0.4815174639225006, -0.8668235540390015]
const viewer = new Viewer({
    rootElement: containerRef.current,
    //cameraUp: up,
    initialCameraPosition: [-1, 4, 6],
    initialCameraLookAt: [0, 0, 0],

});

const orientation = new THREE.Quaternion();
orientation.setFromUnitVectors(
    new THREE.Vector3(0, 1, 0),
    new THREE.Vector3(up[0], up[1], up[2]).normalize()
);

viewer.init();
viewer.loadFile(file, {
    splatAlphaRemovalThreshold: 5,
    halfPrecisionCovariancesOnGPU: true,
    position: [0, 0, 0],
    orientation: orientation.toArray(),
})
    .then(() => {
        console.log('File loaded');
        viewer.start();
    });

Suggested solution:

I dug through the code to look for potential bugs, but came up empty handed. I lose track of the splatMesh.quaternion but my guess is because its a member of a threejs parent class.

Thanks!

Larger file size of .splat files

Hi,
I was wondering why the file size of .splat files are so high (100+mb) compared to Kevin's .splat files(25mb).
Skimming through the codebase, this implementation also discards the spherical harmonics coefficient. Is there any specific reason for the larger file size ?
Thank you ! :)

Not working on android browsers except firefox android

I have tried the online demo and also set up a website for viewing and configured the proper CORS settings in Nginx, as well as implemented HTTPS. The site is functioning fast and well on Windows browsers and iOS. However, it is not working on the default browser (webview) in Android. I checked the 'crossOriginIsolated' variable using dev tools and found that it is always false, regardless of the CORS settings I apply. Interestingly, the site works well on the Firefox mobile browser on the same device.

Upon consulting MDN, I discovered that while the 'crossOriginIsolated' variable is available in webview, the term 'SharedArrayBuffer' is never supported in webview. Additionally, MDN indicated that Chrome on Android supports SharedArrayBuffer, but it still failed to run on Chrome for Android.

Do you have any ideas on how to resolve this issue and get the website running on Android browsers? A fix would be great help and enable a broader range of users. Thanks.

I have an idea about a 3D Gaussian viewer.

I have added camera viewpoint movement in the scene, although it may not be suitable for every model. Perhaps if I can obtain the camera poses from the trained model, it would be possible to automatically handle the camera motion trajectory. What do you think?

Relighting

Any thoughts on how lighting sim could be done in the future strictly inside three js?

[Question] Is there a "common solution" for centring models?

My current workflow is:

• Create in Luma / poly.cam
• Download .ply
• Clean up in Super Splash
• Convert to .splat with GaussianSplats3D
• Upload / implement .splat

The issue I encounter is that the orientation of a splat is obviously treated very differently among software. I am aware of the Y/Z-axis, but in this case the splat is not 90 degrees off (which I would expect, when y and z axis were treated differently), but the splat is completely turned around.

The screenshot shows what I have in Super Splash and the video here:
https://dl.rowild.at/GaussianSplats3D_upside_down.mp4
shows what GaussianSplats3D is doing with the (converted from .ply) .splat.

I am aware that the distance of the camera has to be adjusted in most cases. (Of course I would love to know if that is the case, because the .splat is scaled differently or is there really just a difference ein placing the camera.)
But, as mentioned above: the splat is turned upside-down in GaussianSplats3D – and that does not really seem to be the case, because the z-axis is defined differently.

Not sure, if my thinking is totally wrong and splats work completely different from meshes, or if there is something else off...
I know my way around three.js so-lala and would like to think that, along with the documentation, I can re-define the OrbitControls and the camera so that my splat is handled properly. But it seems that currently these adjustments have to be done on a splat-to-splat basis.

Support for disposal and freeing up client resources

In a context of SPA web apps we need a method of the Viewer to release hogged up resources.
The issue is: If the viewer is integrated into a SPA framework (Blazor in my case), upon navigating to a "page" with no viewer and then back again to the viewer, there will be a significant performance and stuttering issues up to a point of unusability. Resources from each load/visit of the viewer are still hogging clients memory. This is because resources aren't freed since a navigation in a SPA means just an in-place html replace and NOT a full page reload. I checked memory usage upon each viewer initialization, and it keeps adding up but not freeing the memory since SPA frameworks dont know how to do that with .js resources. SPAs only handle their component disposal which has nothing to do with resources that are used up by .js libraries. This will eventually result in memory leak.
Usually .js libraries offer some kind of a destroy or dispose method. Is there a way for the viewer to do the same? If not I would grately appreciate some sort of an implementation in the future release.

Demo page not working

Latest changes possibly caused a problem - will happily process a file from local system; but only shows a black screen for any of the demo items.

Run establish an online demo error

Thanks for your great work !
I have a program

Reproduction Step:

I use the nginx proxy pass port, and add the following configuraions:
add_header 'Cross-Origin-Embedder-Policy' 'require-corp';
add_header 'Cross-Origin-Opener-Policy' 'same-origin';

Acutal Result:

But appear:
DOMException: Failed to execute 'postMessage' on 'DedicatedWorkerGlobalScope': SharedArrayBuffer transfer requires self.crossOriginIsolated .

How can i resolve this program~
Thanks

want accessible mobile page

Your work is a good one. What I would like to ask is Deploying this project to an accessible web page is accessible on the computer side, but not on the mobile (iPhone is ok) side, and it shows the problem of loading all the time.
Do you know where you need to change your source code?

best regards,

Feature: Support large scenes and large file sizes

There is a 2GB size limitation for JavaScript strings. Will there be any support for handling large files in JavaScript, such as file partitioning or other techniques?

Thanks!

`.splat` universal format discussion

Hello! I'm the author of gsplat.js, in which I'm using the splat format as provided in antimatter15/splat

I have opened an issue on splat compression, and I think it would be great if we can have a universal representation, with a consistent header to support different compression methods.

I can replicate your compressed format, but maybe we can open a common format repo with test files, so we can stay on the same page?

What do you think?

Is it possible to modify the 'splatAlphaRemovalThreshold' parameter during the rendering process?

This library is very useful and perfectly meets my needs. I would like to implement a loading animation similar to LumaAI based on this. It seems that they may also control the 'splatAlphaRemovalThreshold' parameter to achieve this, as seen here: https://lumalabs.ai/capture/443a844e-525f-41e1-9750-1dff62782466. I roughly examined the code and it seems that the 'splatAlphaRemovalThreshold' parameter is already passed to the 'loadFileToSplatBuffer' function before rendering, and afterwards, it seems that the parameter cannot be modified to adjust the rendering behavior. Therefore, I would like to know if I have missed anything or if there is any way to achieve this, thanks！

[Help needed] - File loaded, but not displayed

I setup an example here:
https://stackblitz.com/edit/github-ugddhz?file=index.html,src%2Fapp.js

The file is loading, the CORS are working, too. But nevertheless I do not see my file. It is a ply file from poly.cam that I converted with your converter to make it a splat file. But other files are there, too (also in the "local" public folder), and they won't work, either.

Am I doing something wrong?

And is there another place to ask question or can I ask here?

PS: I should add that the converted splat file is displayed correctly on the website demo page, where it is possible to upload a file...

error ，need help

npm run build

[email protected] build
npm run build-library && npm run build-demo

[email protected] build-library
npx rollup -c && mkdir -p ./build/demo/lib && cp ./build/gaussian-splat-3d* ./build/demo/lib/

./src/index.js → ./build/gaussian-splat-3d.umd.cjs...
created ./build/gaussian-splat-3d.umd.cjs in 124ms

./src/index.js → ./build/gaussian-splat-3d.module.js...
created ./build/gaussian-splat-3d.module.js in 64ms
命令语法不正确。

PR

would be great to PR some of those classes to mrdoob/three.js?

How can I successfully run the demo without the CORS settings

Is there any other way to run without setting CORS，Thank you

Offsetting splat

Hello, I am not very familiar with Three.js but have worked with Webgl and was wondering where in the code I might be able to offset the origin to draw the splat in a different location. I presumed it should be in the SplatBuffer where I have tried modifying the position functions to shift the splat. But all I have achieved is some of the guassians being shifted and not all. So I was wondering if you would know off the top of your head where this would take place. I also tried adding a translation matrix buildPreComputedBuffers() and that didn't seem to work. If you could point me in a direction where this would be achievable in your code that would me much appreciated