mml-io/3d-web-experience

(MML) 3D Web Experience

This repository contains packages used to run a web-based, multi-user 3D web experience that supports MML (Metaverse Markup Language).

It can be easily deployed to environments that support Node.js and can expose ports to the internet.

Packages

This repository includes the following published packages:

• @mml-io/3d-web-experience-client
  • Client for a 3D web experience that includes user position networking, MML content composition, MML-based avatars, and text chat.
• @mml-io/3d-web-experience-server
  • Server for a 3D web experience that includes user position networking, MML hosting, and text chat.
• @mml-io/3d-web-client-core
  • The main components of a 3D web experience (controls, rendering, MML composition etc.) that can be extended with other packages to create a full 3D web experience.
• @mml-io/3d-web-user-networking
  • WebSocket server and client implementations that synchronize user positions.
• @mml-io/3d-web-avatar
  • Creates and parses MML documents for avatars (using m-character).
• @mml-io/3d-web-avatar-editor-ui
  • UI components (e.g. parts pickers) for creating avatars.
• @mml-io/3d-web-standalone-avatar-editor
  • An MML avatar editor (using m-character).
• @mml-io/3d-web-text-chat
  • Contains WebSocket server and client implementations for text chat.
• @mml-io/3d-web-voice-chat
  • Client implementation for spatial voice chat.

Main features

• Multiple users can connect to the experience using just a web browser.
• Users can interact simultaneously with the stateful MML documents.
• Easy to deploy and extend with interactive MML content.

Auth Flow

• When the client page is rendered by the server, the server uses a UserAuthenticator implementation to determine if a session should be generated for the incoming http request and if so includes that session token on the client page.
• The client then sends the session token in the first message to the server when it connects via websocket.
• The server can use the session token to authenticate the user and determine what identity (username, avatar etc) the user should have.
• An example implementation of this is provided in the example server, but the interface is extensible enough that a more complex user authenticator can limit which avatar components should be permitted based on external systems.

Running Examples & Iterating Locally

Making sure you have Node.js installed, run the following from the root of the repository:

npm install
npm run iterate

Examples

• example/multi-user-3d-web-experience
  • Once the server is running (see above), open http://localhost:8080.
  • A client and server pair of packages that uses the @mml-io/3d-web-experience-client and @mml-io/3d-web-experience-server packages to create a multi-user 3d web experience that includes MML hosting and text chat.
• example/local-only-multi-user-3d-web-experience
  • Once the server is running (see above), open http://localhost:8081.
  • A client that uses the various packages to create a 3d web experience that only works locally. No server is needed, but there is a server to serve the client.
• example/web-avatar-editor
  • Once the server is running (see above), open http://localhost:8082.
  • An avatar editor that uses the @mml-io/3d-web-standalone-avatar-editor to create and edit MML avatars and a simple Express server that hosts the editor.

3d-web-experience's Issues

User Auth

m-character Support

First person view fixes

Spatial Voice

As originally suggested in mml-io/mml#78 by @DirectiveCreator.

Spatial voice chat should be added to the 3D web experience to allow talking to nearby users.

[Bug]: Shadow Acne caused by lack of normal offset

Describe the bug

Since day one we've been using a scene (not only on this project) with lights with the default values for shadow bias and normalBias (both default to 0) which are often required to prevent shadow artifacts.

That caused issues like the ones in the screenshots below:
(the stripes on the dice)

(same issue on another GLB object with a basic material)

Because the stripes and patterns caused by this bug had different distances between one another depending on the size of the object, its depth on the scene (distance from the character), etc., the pattern sometimes was not noticeable but still affected the colors of everything on the scene.

Solution:
This bug can be eliminated just by introducing a small value to the shadow.normalBias of the directionalLight that acts like the sun on our main scene. A small radius was also added to the shadow to improve the quality of the shadows at the cost of a minuscule and imperceptible distortion of the shadows.

Expected behavior

Shadow acne should be tweaked and eliminated as much as possible for all objects and material types without compromising the quality of the shadows. Please see the before the fix/after the fix in the screenshots below:

(before the fix)

(after the fix)

After applying the fix, it was possible to obtain more fidelity on the color of MML objects used on the scene when there's no tone mapping and other influences on the colors of the scene, as seen in the screenshot below:

How to reproduce the bug

Importing any GLB to our pre-fix scene on 3d-web-experience caused shadow acne.

Link to the code that reproduces this bug

No response

Extra details

Fix (already described in this issue's description) was already applied and will be pushed in my next PR.

[Bug]: Avatar skeleton animation position/rotation conflict

Third person camera wall clipping

[Bug]: raycast on CollisionsManager reporting wrong height to character's LocalController

Describe the bug

When importing vertically stacked structures from GLB models in MML documents, the currentHeight property on the character's LocalController.ts reports the wrong value, causing the getTargetAnimation to return a falsely-positive AnimationState.air state, hence, making the character appear with a falling animation while it's walking down the structure.

This type of structure triggers the bug:

Observable bug behavior:

Expected/correct behavior:

Cause of the bug:
The three-mesh-bvh raycast method invoked from the raycastFirstDistance method on our CollisionsManager class does not return the hits array in the consistent and predictable order we expect, making the 0 index value not to be the expected one (the closest hit point from the ray's origin).

How to solve:
Replacing the raycast method from three-mesh-bvh with the raycastFirst method from the same package guarantees that only the ray hit against the closest mesh in the ray direction will be returned, hence, we get the correct currentHeight property on the LocalController class allowing us to properly identify the right animation state according to the characterOnGround, currentHeight and characterVelocity.y values.

Expected behavior

The currentHeight property's value on the LocalController class should always be the distance from the ray origin (character's object pivot) to the closest mesh in the vectorDown ray direction.

How to reproduce the bug

By importing any structure similar to the one in the provided screenshot (ideally flights of stairs that can be climbed down) and trying to walk down after passing the second flight of stairs.

Link to the code that reproduces this bug

3d-web-experience/packages/3d-web-client-core/src/collisions/CollisionsManager.ts

Line 48 in a697416

const hit = value.meshBVH.raycast(ray, DoubleSide);

Extra details

Fix already implemented and tested will be pushed in my next PR.

Dynamic Environment Map

[Bug] Ambient occlusion disappears away from origin

Standing on moving platforms

[Bug]: Failed to load character avatar

Describe the bug

When deploying the 3d-web-experience to a Web-Server with public IP (and/or Caddy reverse proxy, same behavior with and without proxy, with and without https), the Mesh of the Avatars is not loaded. Note that the avatars are labelled and also collisions work.

Observed behavior
On non-localhost (see https://hackverse.quintinius.xyz), the avatars cannot be loaded. Note that external GLB via <v-model>load fine, refer this MML Document

Error from client-side console, Google Chrome and Firefox have identical behavior:

Obviously, the (cached) GLTF/GLB cannot be delivered to the client. I did not see any CORS warnings or similar. When trying to access the URL, i.e. https://hackverse.quintinius.xyz/3967a9d6-ce92-4ab5-8764-39be57a6d6ec, HTTP 404 is returned, the bodyreads as Cannot GET /3967a9d6-ce92-4ab5-8764-39be57a6d6ec

Expected behavior

The avatar is loaded and rendered. This works on localhost (same behavior when accessing it via local network):

How to reproduce the bug

git clone https://github.com/tbergmueller/mml-hackground/
(this is corresponds to this repository's 6a3b4e0, it's only extended by a docker-compose and some models in the mml-file)

npm install
npm run iterate

OR via my dockerized environment from this commit

docker compose build --no-cache
docker compose up -d

You can play the deployed version on
https://hackverse.quintinius.xyz.

Link to the code that reproduces this bug

git clone https://github.com/tbergmueller/mml-hackground/

Extra details

I tried to deploy on multiple servers, it is currently deployed on a dedicated root server https://www.hetzner.com/de/dedicated-rootserver/ running reverse-proxy via https://caddyserver.com/

The observed behavior is the same across different servers and whether using HTTP, HTTPS, reverse-proxy or not.

Postprocessing chain improvements

[Bug]: Character rotates too slowly in lower frame rates.

Describe the bug

When the excess of complex objects on a scene drops the frame rate to a low value, the character takes too long to change direction due to the way we're rotating the character in the LocalController class.

Cause of the bug:
Our updateRotation method uses a "fixed-rate" lerping method to change the character's model quaternion to the new target value.

Solution:
By establishing a fixed desired time interval to rotate the character, and calculating the angular difference between the current model's quaternion and the new desired quaternion (set from the player's input and correspondent azimuthal angle), we can use the scene's deltaTime to make the character turn consistently on high or low frame-rates.

rotationQuaternion.setFromAxisAngle(this.vectorUp, this.azimuthalAngle + this.rotationOffset);
const angularDifference = this.computeAngularDifference(rotationQuaternion);
const desiredTime = 0.07;
const angularSpeed = angularDifference / desiredTime;
const frameRotation = angularSpeed * this.timeManager.deltaTime;

Method to compute the angular difference:

private computeAngularDifference(rotationQuaternion: Quaternion): number {
  if (!this.model?.mesh) return 0;
  return 2 * Math.acos(Math.abs(this.model.mesh.quaternion.dot(rotationQuaternion)));
}

Expected behavior

The character should rotate with the same lerping factor, in the exact desired target time interval, regardless if we have a low frame rate

How to reproduce the bug

Populate the scene with enough complex objects to drop the frame rate of the scene below 50fps (press p to open the rendering Tweak Pane and look at stats -> FPS), and try moving with the character changing directions.

Link to the code that reproduces this bug

3d-web-experience/packages/3d-web-client-core/src/character/LocalController.ts

Line 174 in a697416

this.model.mesh.quaternion.rotateTowards(rotationQuaternion, 0.07);

Extra details

The solution (already provided in the bug's description) will be pushed in my next PR.

[Bug]: Improper TweakPane setting preventing custom tone mapping to be disabled

Describe the bug

Improper settings on the TweakPane manager here and here prevented the custom tone mapping from ever being disabled on the scene, thus, preventing the user from getting the scene's post-processing interference completely disabled and the color-space linear, to check for color acuity in MML objects.

Expected behavior

With the tweakpane settings as in the screenshot below, the user should get the scene with colors as linear as possible to compare the colors of MML objects and check if they look like they're supposed to (obviously the light of the scene will still interfere with colors a bit due to its interaction with the material used for our m-elements but that's expected. The aim here is to allow the user to disable post-processing interferences).

How to reproduce the bug

1. open the 3d-web-experience;
2. set the renderer tone mapping to custom on the tweakpane (press p to toggle), and set some custom tone mapping;
3. move the renderer tone mapping to the first position (0, no tone mapping);

The custom tone mapping pane blade should disappear and the custom tone mapping should be completely disabled, but it isn't.

Link to the code that reproduces this bug

No response

Extra details

No response

Chat messages above head labels

Web Experience Avatar Customisation

3d-web-experience packaging

Web Experience Chat

Indoor/outdoor lighting handling

Web Experience Demos

Character controller improvements

Separate Dependencies for Web Experience

Avatar Animations