etherealengine/etherealengine

Ethereal Engine - Bringing us together on the open social spatial web. 🤖 🚀 👓 🕹️ 🧑🏿‍🚀

Manifest your dreams on the open social spatial web.

Table of Contents

• Table of Contents
• Background
• Demos
• Project API
  • Library of EE Projects
  • Create your own projects
• Launch Ethereal Engine with Control Center Install - RECOMMENDED
  • Use Metaverse Control Plane Managment GUI
• Advanced Development Install
  • Pre-Requisites
  • Getting Started
• Usage
  • Documentation
• Contributing
  • Let's build it together
  • Join our Discord
  • Sponsorship
• License

Background

Universal.Volumetric.Video.Web.Player.mp4

We (Ethereal Engine) believe that the Metaverse, Web AR, VR, and XR should be easy.

We want a Star Trek future where participating in immersive communication technology and shared simulations should be as easy and natural as making a website.

• Includes a self-deployable platform, UI, worlds, avatars, and games
• Fully customizable for any kind of game, social experience, event or spatial web app
• User management, avatars and inventory with optional blockchain integration (see our Blockchain-in-a-Box repo)
• Social features like chat, groups, friends, blocking and moderation
• Complete world editing and administration
• 2D, 3D and XR Avatars with full inverse kinematics and facial expressions
• Fully networked physics using Rapier Physics
• Voice and video over WebRTC
• Instant login and location sharing with phone number or email
• Modular- Use the engine, server, client, editor and scalable devops infrastructure as needed
• OAuth login with Facebook, Google, Steam, Github or instant login with email or phone
• WebGL client deployable to iOS, Android and desktop
• Free, open source, CPAL licensed

Demos

Guest Demos

• Apartment - CCO from Sketchfab
• Star Station - our dev coommunity hangout
• Globe Concert Hall - collab with WildCapture.io

Host Demos

• Admin Console - Demo Cluster - Request Demo Access on our Discord
• Ethereal Studio - Demo Cluster - Request Demo Access on our Discord

Project API

Library of EE Projects

Create composable plug-ins for Ethereal Engine experiences with the Projects API

Create your own projects

Launch Ethereal Engine with Control Center Install - RECOMMENDED

Use Metaverse Control Plane Managment GUI

Best for simple self hosting, advanced editor content creation, and XRProject Pack Project development.

Advanced Development Install

Pre-Requisites

To install Ethereal Engine locally, the following pre-reqs are required.

• Linux (many of us develop on Ubuntu), Mac OS X, or Windows (we recommend WSL2)
• Node.js v16 or later (we recommend installing via nvm or asdf.)
• C++ (for node-gyp), Python >=3.6 + PIP, make and other build tools, for compiling Mediasoup. Although most of Ethereal Engine is written in TypeScript, it uses a Mediasoup engine for WebRTC conferencing. See the Mediasoup install instructions for more details.
• Docker. (Although Docker is technically optional, running Ethereal Engine requires starting up half a dozen different services, and using Docker Compose will make your life dramatically easier.)
• MariaDB and Redis. (If you're using Docker, containers for MariaDB and Redis will automatically be started up.)

Getting Started

See the Installation instructions for more details.

Usage

Documentation

• General Ethereal Engine documentation
• Auto-generated (typedoc) API reference
• Avatar, Scene & Asset Pipeline
• Why are we building Ethereal Engine?

Contributing

• We expect contributors and community members to follow our Code of Conduct.
• See the Contributing Guide and corresponding wiki entry for more details.
• Version numbering of the Ethereal Engine monorepo follows the Semantic versioning approach.
• We use the Prettier.io formatter for our code style.
• Standard Readme for the README structure.

Let's build it together

We believe that projects like this are extremely complex and difficult, and can only be built when large groups of people work together, out in the open. If you believe that your calling is to build a free, open network that everyone, everywhere can get value from, then you are welcome in our community, and we'll do our best to get you set up.

We are always hiring talented people who want to be leaders in what is to come. Inquire with anyone who seems like they know what's going on and they'll help you find who you need to talk to.

Join our Discord

Sponsorship

License

CPAL - Copyright (c) 2021-2023 Ethereal Engine, formerly known as XREngine by XR Foundation

If you wish to use Ethereal Engine under the open-source CPAL license, attribution is required. Please see attribution guidelines in the LICENSE file. Other licensing options are available, please contact us for more information.

Release History and Author Archive

etherealengine's Issues

Add REST service calls to backend for room manager to integrate with Agones app

https://agones.dev/site/docs/

Chargebee Integration

@mrhegemon has a good idea of what the client wants who's paying for this feature add, so coordinate with him. We are keeping this mainbranch because payments with SSO conneccted to Feathers is already a superhot featureset :)

Configure ICE / STUN / TURN

Running mediasoup in hosts with private IP (AWS, Google Cloud, Azure)
Those environments run virtual hosts with private IP and provide mechanisms to route an external public IP into that private IP. mediasoup implements ICE-Lite meaning that it won't initiate ICE connections but will always act as ICE “server” role.

In order to run mediasoup in those environments (host with private IP and a mapped public IP):

Let HOST_PUBLIC_IP be the external public IP mapped to your host.
Redirect the port range given by rtcMinPort-rtcMaxPort from HOST_PUBLIC_IP to your host.
Of course, also redirect whichever port your application uses for signaling (HTTP/WebSocket) from HOST_PUBLIC_IP to your host.
Use ip: '0.0.0.0' and announcedIp: HOST_PUBLIC_IP when creating a transport.

https://groups.google.com/forum/#!topic/mediasoup/CwsxtmCcrh4

User Relationships / Friends

Purpose of this feature:
We want users to have a basic friend list and ban list

Step 1 -- Users can add and remove other users and friends

Step 2 -- Users can block and mute other users

Step 3 -- Users can view the profile of other users

Step 4 -- Build front end component to reflect all of the API capabillities.

Dynamic Grid Component

We need a dynamic grid component that can be expressed through our object / entity API

Refactor User data to be stored in IdentityProvider and relational table

Currently, users carry around a long list of connected identities Since users will need to be referenced fairly often, and account connections are going to end up with a ton of blank tables for most users, we need to refactor to have an "identity_provider" table which stores oauth, sms and e-mail.

Identity provider hasOne account type, belongsTo user.

Project API (Holds Spoke Projects / Scene Dev Templates)

Spoke - Asset API (high level on resource API)

Avatar API

Add Typedefinitions where missing / TS-ignored

//@ts-ignore
import blobService from 'feathers-blob';
//@ts-ignore
import fs from 'fs-blob-store';
//@ts-ignore
import S3BlobStore from 's3-blob-store';

Let's look through and find any others and get typedefs made.

Look for any other ts-ignore and let's try to remove them and type anything necessary.

Add port and 'secure' to SMTP config

Currently our SMTP config looks like this:

https://github.com/xrchat/xrchat-server/blob/master/src/services/email/email.service.ts

{
    host: process.env.SMTP_HOST,
    secure: true,
    auth: {
      user: process.env.SMTP_USER,
      pass: process.env.SMTP_PASS
    }
  }

Need to:

1) Add a PORT config (default to 2525).

2) Load the secure argument from config (right now it's hardcoded to true. Which is the correct default for deployment, but makes it harder to do Mailtrap.io local testing).

2D Overhead View (Replaces 3D avatars with sprite imposter)

We want to be able to switch between the 3D view and a 2D view. This 2D mode allows phones that can't render everything to completely eliminate all the avatars and just focus on the moving experience.

This will necessitate overhead camera and rotate controls.

We use replace avatars with users' badge icons and clip the camera to just above them so our avatars don't get lost under buildings.

Add models

Users
Has a Contacts list (their friends)
Has many Avatars
Belongs to many Groups (those are ephemeral though)
Contacts
user_id
contact type (friend, blocked, muted etc)
Groups
group_id (string)
user_id (string)
User_role
current_location
XRAvatar
A 3D Model / skeleton, blob. Multiple standard models/formats. There’ll be a list of public templates, but users can have their own / custom ones.
format_type (VRM model, whatever)
Created_by_attribution
owner
name
thumbnail
XRLocationInstance
A particular location has many Instances, for load balancing users. Users can only talk to other people in the same instance. Instances are dynamically allocated; when you join a location, you’re assigned (round robin at first) to a particular instance. (Think “instance dungeon” in an MMO.)
belongsTo Location
belongsTo Scene (references a scene)
location_id - foreign key
XRLocation (XRSpace from the WebXR spec?)
These are the “rooms” that get listed on the front page (or are unlisted, which you can only access via direct link). UI-wise, users “join” a location, but they’re actually assigned to one of the many instances for that location.
Example: VR Event.
is owned by its creator - a User or a Group. (Unless it’s given to somebody else)
hasMany Instances
name (string, unique) - passphrase-style, like Mozilla Hubs
maxUsersPerInstance (number)
access (string) - public/private access etc
XRObject (Not needed right now)
Is an asset file / image / model.
belongsTo User / Group
many-to-many rel’ with Scene (objects are referenced by many Scenes)
object type (static, to start with - “you put it in the room, it stays there.”)
file format
created by
XRScene
“A user made a thing”. A grouping of objects. A scene is assigned to a Location.
Many-to-many rel’ with Locations.
List of objects (with location, position, location. Object id + transform)
(Ideally conforms with Mozilla Spoke)
ScheduledXRLocation
DateTimeRange (turned on / turned off)
Group
Name
XRScene
Payments
Time
User
Type

Remove deprecated deployment args

When our next major deployment is complete and some new server code is up, the following args should be removed, as they will no longer be in use (don't do this until instructed to):

STORAGE_S3_BUCKET_NAME
STORAGE_S3_PUBLIC_VIDEO_PATH
STORAGE_S3_PUBLIC_VIDEO_BUCKET

Add Multer and S3 service connection to Resource API

We want to be able to upload files as resources through our Resource API. Ideally we can handle large files -- so let's use Multer to handle multi-part form uploads and integrate with S3 (or, ideally, a storage service with s3 configuration). Once file is uploaded, store url in resource.

Related to https://github.com/xrchat/xrchat-client/issues/96

Location Manager XRChat Adapter

We want to write an adapter to the location and instance services so that they actually do something with regard to backend logic on our server. This means integrating with the Agones API to gain understanding about users in each instance, number of instances, etc etc. to make decisions on where and how to route users.

Add "All Contributors" style contribution and crediting

https://github.com/all-contributors/all-contributors

We want to encourage everyone on the team to be on the git and feel involved, so let's design a contributors list that let's everyone share some credit for building this thing.

Allow player to spawn new objects

Select objects from our backend or external services (Sketchfab, TurboSquid, Google Poly, etc...)

Party CRUD + UI View

Purpose of this feature:
This is like a "party" system in a video game. User A sends user B a request to join User A's party.

This might be expressed as a teacher and a class full of kids. When the teacher presses the button to enter a new world, it will instruct the kids to go with them into this room.

Step 1 -- Create the service (done)
Step 2 -- Associate with users
Step 3 -- Add to party, join party, remove from party and leave party
Step 4 -- Integrate with gameserver and instances for event messages like joining a new server.

Parties are small, ephemeral groups of users. There must be at least two people in a party for it to exist -- otherwise it is destroyed automatically.

Parties have their own chat, and party members can join each other's worlds. Party leaders can invite and kick party members from the party. When a party leader joins a location, every other party member will see an offer to join with them.

Party members can also join the room of any other member at any time, although the intention is that this feature will simplify helping groups all join the same room together.

Search API

Continuing on Spoke development, integrating Search API to allow Spoke to search for objects from Sketchfab, Google Poly or internal collections. Media search bar should fully integrate with our backend.

We will eventually want to add a search term blacklist -- this will be handled in a separate ticket, but please design in a way where a search term blacklist is not difficult to build in.

Move video rendering to a separate service, possibly do them sequentially

Rendering just 10 videos at once concurrently on the server is taking a very long time, on the order of an hour. It would likely be better to do this outside of the API server so as not to max it out and cause rendering drops if the API is being redeployed.

Not sure if a standalone server/K8s pod, Lambda, or something else would be the best platform.

Also not sure if doing one at a time from a queue is better than multiple ones at once.

One thing we've considered is giving our server a "mode" type -- so that it can run as a media server and only expose those endpoints, or as an API or realtime gamserver, for example.

Set up Error Log Aggregation on server and client

Set up 1) error logging, and 2) stream those logs to an aggregator like Splunk or Loggly.

(Related: Exception monitoring, issue xr3ngine/xr3ngine-ops#16.)

Exception Monitoring and Alerting infrastructure for server and client

Let's go with a hosted solution first (the free tier one) -- Sentry.io. (Second option, if Sentry doesn't work out: Rollbar).

(Related: Error logging, issue #309.)

Change email subject line to come from config

(This applies to any auto-generated email, but specifically to the Email Login Link emails.)

Email subject line reads "Magic Link to sign in",

Expected: language to mirror the log-in screen and identify the community from which the email is being sent ex: "Login to KaiXR.com"

Right now the email subject line is hard coded, let's instead draw from config.

Fix remove/deletion errors

When user starts server, "failed to remove" errors appear on start (after ~ 20 seconds). This is because we need "use multi" option but don't have specific sequelize definitions for our junction tables, so they are preventing a multi-delete on the sync-seed in dev mode.

We need to figure out how to enable "multi" for models that aren't explicitly defined so that these errors go away

Entity API

What does this feature do?
We want to be able to consume objects and store them, and then make them available when requested.

Actual objects and resources will be uploaded to S3 via our storage API. @barankyle has done this.

First step: Add CRUD for objects. Owner is 'server' (default) unless it is attached to a userID
We already have an Objects API. Update the objects.model.ts to have a list of resources
Feathers automatically makes CRUD -- read documentation to override if necessary

class Object(){
id: number (unique)
owner: relation to user (empty for server)
created: dateTime
modified: dateTime
resource: {type, url}
}

Second step: Integrate with Storage API
AddResourceToObject ({objectId, type, url}) -- Upload new resource and attach it to the object

Third Step: Component on front end that lets user upload a GLTF or GLB (POST)

Fourth step Add image upload, OBJ and FBX upload (POST)

Fifth Step Form shows object, user can edit or replace resource (UPDATE) to delete (DELETE)

Sixth step: Integration with XRPackage. Upload and download XRPackages

Please check out:
https://github.com/webaverse/xrpackage

Add Faker, seed db with data

Add package update manager

This will keep our dependencies up to date (as PRs) and let us stay on the latest stuff in a way we can validate, test and roll back.

Portal To Other Location

The goal here is a portal that can take a user from one location to another, which implicitly means that it will also be taking them from one instance server to another. This is primarily a front-end component, and the portal-as-a-link concept has been demonstrated, but I think it'd be cool if we didn't something that didn't require reloading Aframe.

My proposed flow:

1. User taps and holds to charge portal for 2 seconds (to prevent accidental button taps)
2. Void fades in as all objects in the scene fly directly away from the user
3. Objects unload, peers disconnect
4. New peers connect, new scene loads
5. Void dissolves back out

The void be matrix loading construct, could be a warp gate thing, whatever, have fun with it. Maybe sound?

Agones API Integration

This issue has been changed to cover integration of Agones with the Location / Instance APIs so that we can dynamically get IP addresses to game servers from the k8s cluster

Upgrade to Feathers.js v3

Upgrade server to Feathers.js version 3.
Document any upgrade or migration issues.

Let's see if we can fix these warnings on install:

npm WARN deprecated @types/[email protected]: This is a stub types definition. graphql provides its own type definitions, so you do not need this installed.
npm WARN deprecated [email protected]: Feathers v3 is out and this module has moved to @feathersjs/commons. See https://docs.feathersjs.com/migrating.html for more information.
npm WARN deprecated [email protected]: This module has been integrated into @feathersjs/commons.
npm WARN deprecated [email protected]: Feathers v3 is out and has moved to @feathersjs/errors. See https://docs.feathersjs.com/migrating.html for more information.
npm WARN deprecated [email protected]: Feathers v3 is out and this module has moved to @feathersjs/commons. See https://docs.feathersjs.com/migrating.html for more information.
npm WARN deprecated [email protected]: core-js@<3 is no longer maintained and not recommended for usage due to the number of issues. Please, upgrade your dependencies to the actual version of core-js@3.
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated [email protected]: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.)

Advanced User Settings

GOAL:
A full stack solution -- API and front end menu -- so users can configure their avatar icon, connect and disconnect accounts and adjust settings.

(Should user settings and user profile be separate / separate related models)

User Badge Menu and Profile Component
In the front end, when you log in, there is a user badge and icon
When you click on the badge, it has a dropdown menu: profile, contacts, logout

If user clicks on profile: open a modal window where user can upload a profile photo (create and embed UserIcon component), edit their settings (create and embed UserSettings component) and connect additional accounts (embed existing "connect accounts" component our other engineer is building)

Details of settings require more review.

Planned so far.

User Icon Component
Upload / replace user icon
This is a "resource" of type "user-icon"
Add resource type to seed-config
Copy /upload-video pattern for user icon upload endpoint

User Profile / Settings Component
Enter room muted: true/false
Microphone volume: 0-100
Audio playback volume: 0-100
Visual Quality: High / Low
Use anti-aliasing: true/false

Add tests to verify that all APIs work on npm test

Right now our services register that they work and exist, but that's it. We should add unit tests to validate the models, especially foreign key dependencies, via CRUD.

Connect / Disconnect Identity Providers

Create a react UI component that allows current user to connect and disconnect email, sms and oauth providers from their account. This will go into the user profile component that @AyushiMastro is making.

Load testing

Synthetic load tests on

static assets

one instance

instance provisioner / agones k8

geo diverse traffic

Admin Realtime Controls

API controls to perform admin operations on users

Kick / Ban / Mute
Move User to Location
Move User to Instance

Advanced CDN and content process / store features.

We need a UI component for a user to upload an image, video or 3D model to our backend. This will interact directly with our low-level resource API and upload multi-part form data.

Add Storage API

Storage API allows other services to save and load static files and resources without having to implement a specific storage strategy or API. Which storage type is used should be stored in server config (xrchat-server/config/default.json)

Storage API should have two default storage adapters: local and S3. Local is default, and stores files to a public or static directory.

Related to #64

Create VR HUD

YouTube media cacher

Cache video to S3 from YouTube
This will need to be fleshed out -- we need to route content requests through to check if it already exists in our cache.

Scene API

Add Auth Hook To All Routes (where appropriate)

Skipping auth makes API development a little bit faster since you don't have to juggle tokens or worry about setup, so we should implement this toward the beta milestone.

Identify all points of risk, add auth hooks to routes and define a clear public API surface. Add auth hooks to all services, or set all endpoints to require auth by default and whitelist public routes.

Add Networked Physics Engine

In progress?
https://github.com/HydraFire/ecsy-physics-system

We need a physics solution that can be used by all of our networked minigames and locations.

Please use the emscripten port of
NVIDIAGameWorks/PhysX#238

Alternate
ammo.js for physics here -- https://github.com/kripken/ammo.js

Node group clusters for all node types

Working with agones, redis, mediasoup, etc.

https://docs.feathersjs.com/cookbook/general/scaling.html#cluster-configuration

Location API + Location CRUD / Manager UX

Goal of this feature:
To route a user from one URL out to one of many different game servers in out Kubernetes cluster.

Two services
location -- creation and deletion of a location, which is a set of instances. Each instance is a game server. Our game servers are manged by Agones -- http://agones.dev

instance -- creation and deletion of instances tied to a location. Each instance is mapped to only one location at a time

Step 1 -- Build the API service and data layer with no API connections

Step 2 -- Add an adapter interface and that be extended by backend adapters for different APIs.

Why? We are currently using a FOSS platform called Mozilla Hubs, and we want to build this feature to work with their stack.

Step 3 -- Build an adapter for our own gameservers. We will need to coordinate with devops and our realtime team on lifecycle hooks.

When deleting a location, we should shut down all active instances for that location as well

Location Model:
id, name, instances (relational to the instance service)

Instance model:
id, location, url, currentUsers, maxUsers

Step 4 -- Add front end components that tap into these APIs.

More stuff, mostly from before the ticket was rewritten:

Location Manager can create a location
Has two adapters: Hubs and XRChat
Optionally can supply a number of instances to allocate to location
For Hubs, this will tap their API to create rooms and return room addresses
For XR3, will tap the Agones API

Set up hooks to poll # of users (Hubs doesn’t have this)
Logic to route users to non-empty, non-full rooms first
Init instance server with information about scene if it isn’t already part of location
Instances with low users can ask Location Manager to reroute users to more full room
When a user goes to location endpoint, redirect them to address of realtime server

Pass uploads service test

Currently skipping the uploads service test. Please fix it, since it's currently failing.

User Profile

Purpose of this feature:
Users can take a photo of themselves and add a brief bio.

Refactor public-video model, merge it into resources model.

Our public-videos Model (and database table) currently looks like:

    {
   	 "original_title": "My Video",
   	 "title": "The Video Title to be Displayed",
   	 "link": "https://www.youtube.com/watch?v=abc123",
   	 "description": "Some description",
	“thumbnail_url”: “ “something.jpg”,
        "production_credit": "...",
        "rating": "",
        "categories": "...",
        "runtime": "..."
        "tags": "..."

    }

This needs to be moved into the Objects model, since that's exactly what Objects were intended for - external assets with a title, link, description, thumbnail.

(This will be a stepping stone to also storing Scene thumbnails in objects, so we can re-use the video grid component for them.)

(Also, drop the original_title field - we don't need it.)

Blob Storage / Volume Service With S3 Adapter

Webaverse Integration

WHY?
We want to integrate Avaer's webaverse stack.

Step 1: Research webaverse
Step 2: Look at xrpackage (npm install xrpackage)
Step 3: Look at VRM model format
Step 4: Write document on best way to integrate so we can consume and produce XRPackages with content from our API
Step 5: How should this affect / alter our API?
Step 6: Integrate xrpackage and add endpoints to backend