mozilla/hubs

Duck-themed multi-user virtual spaces in WebVR. Built with A-Frame.

Mozilla Hubs

The client-side code for Mozilla Hubs, an online 3D collaboration platform that works for desktop, mobile, and VR platforms.

Learn more about Hubs

Getting Started

If you would like to run Hubs on your own servers, check out Hubs Cloud.

If you would like to deploy a custom client to your existing Hubs Cloud instance please refer to this guide.

If you would like to contribute to the main fork of the Hubs client please see the contributor guide.

If you just want to check out how Hubs works and make your own modifications continue on to our Quick Start Guide.

Quick Start

Install NodeJS if you haven't already. We use 16.16.0 on our build servers. If you work on multiple javascript projects it may be useful to use something like NVM to manage multiple versions of node for you.

Run the following commands:

git clone https://github.com/mozilla/hubs.git
cd hubs
# nvm use v16.16.0 # if using NVM
npm ci
npm run dev

The backend dev server is configured with CORS to only accept connections from "hubs.local:8080", so you will need to access it from that host. To do this, you likely want to add "hubs.local" and "hubs-proxy.local" to the local "hosts" file on your computer:

127.0.0.1	hubs.local
127.0.0.1	hubs-proxy.local

Then visit https://hubs.local:8080 (note: HTTPS is required, you'll need to accept the warning for the self-signed SSL certificate)

Note: When running the Hubs client locally, you will still connect to the development versions of our Janus WebRTC and reticulum servers. These servers do not allow being accessed outside of localhost. If you want to host your own Hubs servers, please check out Hubs Cloud.

Documentation

The Hubs documentation can be found here.

Community

Join us on our Discord Server or follow us on Twitter.

Contributing

Read our contributor guide to learn how you can submit bug reports, feature requests, and pull requests.

We're also looking for help with localization. The Hubs redesign has a lot of new text and we need help from people like you to translate it. Follow the localization docs to get started.

Contributors are expected to abide by the project's Code of Conduct and to be respectful of the project and people working on it.

Additional Resources

• Reticulum - Phoenix-based backend for managing state and presence.
• NAF Janus Adapter - A Networked A-Frame adapter for the Janus SFU service.
• Janus Gateway - A WebRTC proxy used for centralizing network traffic in this client.
• Janus SFU Plugin - Plugins for Janus which enables it to act as a SFU.
• Hubs-Ops - Infrastructure as code + management tools for running necessary backend services on AWS.

Privacy

Mozilla and Hubs believe that privacy is fundamental to a healthy internet. Read our privacy policy for more info.

License

Hubs is licensed with the Mozilla Public License 2.0

hubs's Issues

Switch screen sharing to P2P

In order to sidestep a variety of issues, we should change the screen sharing implementation to P2P. This will require negotiating a RTC peer connection with everyone in the room (as well as dealing with join/leaves.)

Virtual joystick center position is offset from visual representation

The actual center of the virtual joysticks are nearer to the corners of the screen than the visual circles.

Implement better 2d phone controls

the current "dual axis" controls on mobile work but are not ideal. we should implement the control scheme where single finger drag orbits camera and dual finger pinch-to-zoom moves forwards and backwards.

Mobile friendly water for environment

Safari WebRTC audio not working

Probably should get this fixed before MVP launch due to the surface area of the platform, etc.

Implement desktop-local screen sharing UX

TBD what exactly this means, but we should have a sane UX for screen sharing that doesn't require a hotkey. I think the model where:

• You see one copy of your local desktop that is private
• There is a specific place in the environment for beamed screens
• You can beam/share your screen

Is a good start. Not needed for MVP launch.

Disable microphone on mobile when navigating away from app

Currently on mobile, the microphone is left enabled when you navigate away from the app. This is good behavior in apps where you are having a conference call and multitasking, however I don't think its the same for our app.

We should disable the microphone or disconnect the user from Networked AFrame when navigating away from the application on mobile. We can probably use window.onblur on mobile only.

This is maybe desirable on mobile, but definitely not on desktop when screen sharing.

Client auto-reconnect and repositioning

If the client's internet connection dies, it should reconnect to Janus and respawn the user, etc.

Asset Versioning and LODs

Assets currently use a high quality and low quality url. There also isn't a great way to version them.

@gfodor @netpro2k @j-conrad @brianpeiris and I talked a bit about this in this slack thread

Here are some highlights:

Greg proposed an asset manifest file format that includes both versioning and LOD information:

http://duck.mozilla.com/assets/avatars/robot.XXX
http://duck.mozilla.com/assets/avatars/robot-0.1.0.XXX
{
  "name": "robot",
  "version": "0.1.0",
  "layers": [
    {
      "name": "base",
      "bundles": [
        { id: "robot-1", url: "https://duck.mozilla.com/assets/robot-desktop-0.3.4.glb" },
        { id: "robot-2", url: "https://duck.mozilla.com/assets/robot-mobile-0.3.4.glb" }
      ]
    }
  ],
  "hints": {
    "mobile-gen-1": ["base.robot-1"]
  }
}
http://duck.mozilla.com/assets/scenes/conference-room.XXX
http://duck.mozilla.com/assets/scenes/conference-room-0.2.0.XXX
{
  "name": "conference-room",
  "version": "0.2.0",
  "layers": [
    {
      "name": "environment",
      "bundles": [
        { id: "room-1", url: "https://duck.mozilla.com/assets/room-desktop-0.3.4.glb" },
        { id: "room-2", url: "https://duck.mozilla.com/assets/room-mobile-0.3.4.glb" }
      ]
    },
    {
      "name": "spawn-points",
      "bundles": [
        { id: "room-spawn", url: "https://duck.mozilla.com/assets/room-spawn-0.1.4.glb" }
      ]
    },
    {
      "name": "duck-interactables",
      "bundles": [
        { id: "room-ducks-1", url: "https://duck.mozilla.com/assets/room-ducks-desktop-0.0.4.glb" },
        { id: "room-ducks-2", url: "https://duck.mozilla.com/assets/room-ducks-mobile-0.0.4.glb" }
      ]
    }
  ],
  "hints": {
    "mobile-gen-1": ["environment.room-2", "duck-interactables.room-ducks-2"],
    "far-draw-distance": ["environment.room-2"]
  }
}

Dom proposed using Microsoft's LOD glTF extension

Brian and I liked the idea of using .gltf instead of .glb for situations where you have shared assets. Ex. our avatars that only swap one texture between variants. Everything that is shared between the assets will be fetched from the cache.

We decided to delay working on this until after we have a few finished environments and avatars.

Add Google Analytics

For initial OKR metrics tracking and error handling, we should transition to using Google Analytics (which is sanctioned by Mozilla.)

Nametags are hard to read against the background environment

Make them lighter or add a backing plane

Experimentation spike on ultrasonic data xfer, etc

We need to land on the proper mechanism(s) for doing device pairing for standalone devices like oculus go, or linking your desktop to your mobile for cases like screen sharing. Some relevant links:

https://developers.google.com/nearby/messages/overview
https://youtu.be/Acdu2ZdBaZE
https://youtu.be/1a0wII96cpE
http://smus.com/ultrasonic-networking/

Disable microphone when navigating away from the app

Currently on mobile, the microphone is left enabled when you navigate away from the app. This is good behavior in apps where you are having a conference call and multitasking, however I don't think its the same for our app.

We should disable the microphone or disconnect the user from Networked AFrame when navigating away from the application on mobile. We can probably use window.onblur on mobile only.

zombie avatars remaining on leave

when users leave the room, sometimes their avatars remain. likely call stack: https://mozilla.slack.com/files/U723XJLDP/F835ECFCY/image.png

Get smoke test back up and running

The smoke test bots are no longer working.

Complete hand gesture work on vive and rift

Not sure but I believe hand gestures still aren't fully implemented on vive. Please close if not the case!

Additional environments for MVP

Placeholder for discussion around new environments for MVP

Avatars sometimes do not disappear when people leave

Zombie avatars remain in the room.

Microsoft Edge hangs on connect

No errors are emitted. However it looks like its getting hung up in naf-janus-adapter's createPublisher() method.

Avatar selection

Need to be able to pick an avatar in the UI.

Desktop sharing video freezes

Audio check usability improvements

The current audio check should be improved:

• The mic level should be more sensitive
• The user should be able to click to play audio tone or see countdown
• Consider removing circle

Navmesh based collision detection for locomotion + teleporting

We need to make sure collision is set up properly for the 2-4 environments we launch with.

Allow creation of and environment selection for a new room

Also includes reticulum, implement the react flow for creating a room and choosing the environment.

Migrate service to proper domain + prod cluster

Still TBD based on marketing etc.

Reduce Size of Assets

The .glb assets can use compressed .jpg textures with 80% quality to significantly reduce file sizes and initial scene load times.

Interactables/templates need to be incorporated into the GLTF bundle system

Pending #85 being completed we need to add interactables as a "layer" to the environments since they need to be laid out properly. The main thing to decide here is how to deal with template tags for the bundle, and this will also be dependent upon the GLTF component work since we will need to attach those components.

Avatar Name Tag Improvements

We should tune up the nametags to look better and be easier to read.

frame drop on user join

We noticed big frame drops when users joined during demo

Yarn install fails on WSL

see: microsoft/WSL#2448 for issue and workaround

Avatar spawn point randomization

All avatars currently spawn at origin

Janus SFU load test/capacity planning

We should simulate load to determine our per-box capacity for the janus SFU for a few EC2 instance types (at least if the one we have now tips over easily.) Goal should be 1000 simulated CCU in 83 rooms of a dozen people. We will probably need to get metrics on the box first about message delivery latency, bandwidth usage, etc.

Avatar heads do not appear in Firefox on Android

GearVR entry flow goes to wrong URL for `vr_entry_type=gearvr`

Reproduce by just doing gearvr flow.

No audio output on iOS Safari

We also seem to be connecting to janus twice somehow, which may lead to the audio output issue.

Positional audio broken in Chrome for Android (Daydream)

Cannot pick up objects on Vive

Right now superhands seems to work only with touch controllers.

Cannot hear user (Subscriber for client: x does not exist.)

Support GearVR gaze controls

We should ensure if you don't have a GearVR controller that you can use the touchpad/gaze cursor. (However up this should be timeboxed to a day or two's efforts for now if it ends up being particularly difficult to support.)

Hand models should be invisible unless controllers are connected.

Currently your own hand models as well as every other avatar's hand models are visible despite them not having tracked controllers. These models should be invisible until the tracked controllers are active.

Investigate Video SDP Send Parameters on Chrome for Android

Error seen on Chrome for Android 52:

Failed to set remote video description send parameters..

This occurs on this line: https://github.com/mozilla/naf-janus-adapter/blob/master/src/index.js#L245

SDP sent by Janus:

v=0
↵o=- 1511245988964004 1 IN IP4 54.67.121.231
↵s=Janus session
↵t=0 0
↵a=group:BUNDLE audio video
↵a=msid-semantic: WMS janus
↵m=audio 9 RTP/SAVPF 111
↵c=IN IP4 54.67.121.231
↵a=sendonly
↵a=mid:audio
↵a=rtcp-mux
↵a=ice-ufrag:r58T
↵a=ice-pwd:w0+1h6bBhREj4nDqKLrtKB
↵a=ice-options:trickle
↵a=fingerprint:sha-256 40:F7:B1:3D:F9:1E:04:B8:FF:5F:FF:5F:25:75:EB:C0:ED:F3:25:06:49:1F:AE:B4:E7:7A:BE:69:2F:DA:CA:B2
↵a=setup:actpass
↵a=rtpmap:111 opus/48000/2
↵a=ssrc:3516440168 cname:janusaudio
↵a=ssrc:3516440168 msid:janus janusa0
↵a=ssrc:3516440168 mslabel:janus
↵a=ssrc:3516440168 label:janusa0
↵a=candidate:1 1 udp 2013266431 54.67.121.231 47417 typ host
↵a=end-of-candidates
↵m=video 9 RTP/SAVPF 107
↵c=IN IP4 54.67.121.231
↵a=sendonly
↵a=mid:video
↵a=rtcp-mux
↵a=ice-ufrag:r58T
↵a=ice-pwd:w0+1h6bBhREj4nDqKLrtKB
↵a=ice-options:trickle
↵a=fingerprint:sha-256 40:F7:B1:3D:F9:1E:04:B8:FF:5F:FF:5F:25:75:EB:C0:ED:F3:25:06:49:1F:AE:B4:E7:7A:BE:69:2F:DA:CA:B2
↵a=setup:actpass
↵a=rtpmap:107 H264/90000
↵a=fmtp:107 profile-level-id=42e01f;packetization-mode=1
↵a=rtcp-fb:107 ccm fir
↵a=rtcp-fb:107 nack
↵a=rtcp-fb:107 nack pli
↵a=rtcp-fb:107 goog-remb
↵a=ssrc:1447368931 cname:janusvideo
↵a=ssrc:1447368931 msid:janus janusv0
↵a=ssrc:1447368931 mslabel:janus
↵a=ssrc:1447368931 label:janusv0
↵a=candidate:1 1 udp 2013266431 54.67.121.231 47417 typ host
↵a=end-of-candidates

It's having an issue with the video part, so it doesn't like something in or after this line m=video 9 RTP/SAVPF 107.

This SDP is created by Janus here: https://github.com/mozilla/janus-plugin-sfu/blob/master/src/lib.rs#L437

Add user blocking

Users should be able to block others based upon IP address

Need to hide nav mesh in environments

Moving the GLTF cliff environment to GLTF bundles (via #85) resulted in realizing a few A-Frame components are attached to the a-gltf-entity's involved and there is no way to represent them in the bundle format (nor likely should there be a way to do so.) These include the layer: reflection component on the cliff vista, the visible:false on the nav mesh, and the disabling of ar in the xr components. These will need to be rolled into the GLTF files and whitelisted as inflatable components.

https://github.com/mozilla/mr-social-client/blob/master/src/room.html#L240

Audio from Firefox for Android is distorted

Fix lerping algorithm on objects, add nested lerping support

It looks like we have a regression with lerping in the client. Interactables and avatars are moving in a choppy, non-interpolated way.

Client just stalls at gUM if device does not have an enabled microphone

If all your recording devices are disabled or unplugged, the client just stalls. We should be able to detect this condition and continue after notifying the user

Add a room size cap

We should limit room capacity to some reasonable amount. (Proposal: 100)

Unsure if this should be server or client enforced.

Design cleanup and Reactification of 2D HUD, blocking, teleporting

The 2D HUD work was implemented before we had a react component tree for the main UI flows -- it should be updated to use React. The name of the user should be read from the store and the mic level from the micLevel state from the root of the component hierarchy.

Also a design pass needs to be done to unify the look-and-feel of the 2d HUD with the entry flows.

Add mute control on Daydream

There is currently no way to mute yourself on Daydream.

Skybox for environment

We have a few options for this:

1. Create a component that uses THREE.Sky to create a skybox at runtime and render to a cubemap. This could be an awesome component for ourselves and the AFrame community.

2. Find a suitable skybox and use aframe-cubemap-component.