exokitxr/exokit

Native VR/AR/XR engine for JavaScript 🦖

Exokit

🕶️ Native VR and AR engine for JavaScript 🦖

Discord — Twitter — Email List

Examples

Find more examples here and on YouTube.

Overview

This project enables developers to build XR experiences using the same code that runs on the web. Exokit engine is written on top of Node and emulates a web browser, providing native hooks for WebGL, WebXR, WebAudio, and other APIs used in immersive experiences.

👓 Exokit currently targets the following platforms:

• OpenVR Desktop VR (Steam compatible)
• Oculus Desktop (Oculus Rift/Rift S)
• Oculus Mobile (Oculus Quest/Go, GearVR)
• Magic Leap
• iOS ARKit *
• Android ARCore *
• Google VR (Daydream / Cardboard / Mirage Solo) *
• any XR device, start a pull request with a native binding if it isn't listed here! *

* not supported yet

🔌 Since Exokit supports anything that runs on the web, it powers experiences built with:

• Three.js
• Unity
• Pixi.js
• Babylon.js
• A-Frame
• Custom WebGL frameworks
• WebAssembly, TypeScript, and any language that transpiles to JavaScript

👍 Why Exokit?

• You want your WebGL/WebXR to run fast.
• You want the hot new web APIs.
• You want to add your own integrations -- including native -- into a browser environment.
• You want a lightweight browser as a hackable node module.
• You want to combine the web with a 3D engine like Unity.

👎 Why not Exokit?

• You're looking for a traditional "web browser".
• You don't care about 3D or mixed reality.
• You're looking for strict and/or legacy standards support.

📖 Manifesto

• The future is immersive. The web is the best application platform. Javascript is the best ecosystem.
• Content should be hardware agnostic. Tomorrow will have different hardware. VR and AR should be compatible.
• It's not possible to do both 2D and 3D well. We don't do 2D. We can use an external 2D browser.
• Use your favorite game engine. Exokit is not a game engine.
• Legacy browser design choices don't make sense in XR.
• Exokit empowers and connect apps, even (especially) if they aren't designed to cooperate.
• Apps should run in "reality tabs", layers of reality that blend together.

Quickstart

Desktop

Download for current OS

Run a WebXR site (desktop)

exokit https://aframe.io/a-painter/ # start A-Painter in Exokit

Magic Leap

Download for Magic Leap

Run (Magic Leap device)

mldb connect 192.168.0.10:1131 # mldb needs to be connected; see MLDB documentation
mldb install -u exokit.mpk # downloaded or built package
mldb launch com.webmr.exokit -v "ARGS=node . file:///package/examples/hello_ml.html" # or URL to load
mldb log exokit:*

Architecture

Exokit is a Javascript Node.js module.

C++ bindings hook into WebGL, WebXR, Magic Leap, Leap Motion, and various other device APIs.

const {window} = require('exokit')() to get an immersive browser in another project.

Exokit runs on Windows, macOS, Linux (x64), Linux (ARM64), and Magic Leap (ARM64).

Web API support

• HTTP/S
• HTML5
• <script>
• DOM
• WebGL
• WebXR
• WebVR
• WebRTC
• Canvas2D
• WebSocket
• Web Workers
• <img>, <audio>, <video>
• WebAudio
• Keyboard/Mouse events
• Gamepad API
• <iframe>
• HTML layout
• HTML rendering
• CSS
• Legacy APIs

Hardware bindings

• OpenGL
• OpenVR (Steam VR)
• Oculus
• Oculus Mobile
• Magic Leap
• Leap Motion

Local Development

See full building instructions in BUILDING.md.

git clone https://github.com/exokitxr/exokit.git
cd exokit
npm install
node . <url> # or node . -h for home

Debugging

Uses ndb.

npm run debug

Then in the console, input:

let window = await require('./src/').load(yourUrl);

Now you have a handle on the window object as you test your application, and you can set debugger breakpoints, inspect memory, profile CPU, etc.

Flags

• --xr webvr Makes exokit WebVR mode regardless of the webpage. If exokit is always opening in pancake mode you can use this to try to fix that.

Stay in Touch

• Join our Discord for hanging out.
• Follow @exokitxr on Twitter for updates.

Contributing

Get involved! Check out the Contributing Guide for how to get started.

License

This program is free software and is distributed under an MIT License.

exokit's Issues

Add link traversal support

Basically vrdisplayactivate events need to be emitted when you enter a site.

We actually already have a rough implementation of the emit, but we should clean it up to only emit once, at the right time, and preferably only when the HMD is actually nudged in the first place. I think OpenVR supports querying for this.

Additionally, it's worth thinking about how this should work with our reality tabs (tm :P) interface, where multiple sites are opened at once.

Chromium 2D in 3D

We don't do 2D rendering in the Exokit core, and we probably never will. The future is 3D.

However!

The world needs to catch up, and most of the web is still 2D. So a common request from Exokit is to render 2D sites as a plane. While we don't want to get into implementing the 2D part of the web ourselves, it's not too hard to use another 2D browser engine like Chromium and frame it as a plane in 2D.

We've done this before in Zeo, so we know it's doable. Basically we just need to open a true Chromium instance and feed the framebuffer through to Exokit's GL as a texture. With something like Puppeteer and simple plan intersections emulating mouse/touch events, we would get pretty close to having a 2D browser environment in our 3D browser environment 💻.

Warts

• We need to figure out the web API exposed here. The regular web doesn't normally support this, but some of the extension apis provide inspiration for what it could look like.
• We need to support Window, OSX, and Linux. Ideally we use a library for doing this that's OS-agnostic.
• We would need to ship a build of Chromium and keep it up to date. This should be automated, not vendored into our repositories. We have two CI systems that could help.
• For performance, we probably don't want to use the CPU for this to submit a texture every frame, but instead share the framebuffer.

Oculus HMD testing

We don't have an Oculus headset in the office right now, though it should be supported via OpenVR.

It would be great if a volunteer could check if the basics work in the Oculus VR environment, and file any gotchas/bugs to the main issue tracker on Github.

OSX: close app when home window context is closed

It we don't do this, the OSX app will feel un-OSXlike and hang around in the taskbar.

Spice up the README.md

Exokit does a lot of stuff, such as Magic Leap, Leap Motion, and more. It also runs a ton of VR sites, faster than other browsers. None of this is really mentioned anywhere, and there are no pretty pictures of it in the README.md. Let's fix that!

Basically, it would be great if our frontend content like the README.md had a lot more meat 🍖. This is a fairly open-ended issue.

Avatar hands in exokit home env should not cover face

This is because controllers emulation has the controllers up to the eyeballs. There should be a decent compromise with the skins.

File uploads from VR?

Zeocraft blocker texture is flipped (OSX)

See bottom. Seems fine on windows.

Leap Motion API + docs

We should figure out what the Leap Motion API looks like, as exposed to the use sites.

Ideally we don't stray too far from WebVR/WebXR/device API. Whatever we do should be thoroughly documented on http://docs.webmr.io 📖

ArrayBuffer species is incorrect after fetch

C:\gh\exokit>node .
> fetch('https://mdn.github.io/fetch-examples/fetch-array-buffer/viper.ogg').then(x => x.arrayBuffer()).then(x => { console.log(x instanceof ArrayBuffer); })
Promise { <pending> }
> false

Leap Motion examples

We already have Magic Leap + Leap Motion APIs in the code 🎉. However, what we're lacking is site examples showcasing the awesomeness within.

There's some of this stuff in our fork of THREE.js.

In particular;

• Exokit home supports Leap Motion hands
• Magic Leap - R2 vs BB8

We could use more examples, as well as a structured place to publish them. Either a site or a dedicated repository.

Document CI flow

We currently use both Appveyor and Travis CI (OSX, Linux) for building, verifying, signing, and publishing our releases.

The releases are published to Github and that is where they are pulled from when a user hits https://get.webmr.io/.

We need to document this flow on http://docs.webmr.io/. Or rethink it if there's a better approach!

VR site fixes

Tracking issue for https://docs.google.com/document/d/1GjcJ7AdyRtIMM_BmfJtD9RhlbttmfstKexHO8JL1-VQ

OpenVR does not detect HMD

CC @shawwn to provide repro steps.

What is this returning?

OSX: bad viewport size

(Notice it's ~2x what it should be.)

Re-enable <video> support

Exokit supports <video> tags via its implementation of HTMLVideoElement.

These are backed by the nativeVideo binary binding in deps.

Testing example:
https://threejs.org/examples/?q=vide#webgl_video_panorama_equirectangular

A related nice-to-have:
https://threejs.org/examples/?q=vide#webgl_materials_video_webcam

But this probably would be more complicated, to acquire the webcam feed via getUserMedia.

Use lerna + yarn workspaces

The current best solution for a multipackage monorepo is lerna via Yarn Workspaces.

Add incremental builds

Would be nice for development if npm install built only changed files.

WebGL2 support

Exokit currently only supports the webgl context. WebGL2 adds a lot of nice stuff to that API.

Worry not! This actually is much easier than it sounds; we already use straight up OpenGL 3.2 for out WebGL backing, which is already a superset of what we need for WebGL2. So implementing WebGL2 would be mostly a matter of exposing/defaulting functionality in Javascript per the spec, rather than implementing much native code.

We should try to follow the WebGL/WebGL2 exposure rules in the spec as much as possible.

Cannot run w/ space in install path

It's seems to be an issue on both Windows (install to C:\Program Files) and OSX (App with space.app).

Should be easy to verify and/or fix! The two files which probably contain the affected code are index.js and core.js.

Merge platform-specific repos

We currently support 5 platforms, with platform-specific repos;

https://github.com/modulesio/exokit-windows
https://github.com/modulesio/exokit-macos
https://github.com/modulesio/exokit-linux
https://github.com/modulesio/exokit-android
https://github.com/modulesio/exokit-ios

There are also a couple of helper repos:

https://github.com/modulesio/exokit // this one, javascript/DOM parts
https://github.com/modulesio/exokit-bindings // native bindings

But there's no good reason to have these hanging around long term -- most of the code is exactly the same, on all platforms, even the native parts. Plus, it would be nice to just npm i exokit anywhere.

This mainly requires figuring out:

• the npm install build
• dependency downloads and/or bundling

Running

How would I run this normally, I downloaded the git and when I do npm run it doesn't work I did install the dependencies. please help i'm in the slack

docs.webmr.io improvements

There's some delicious low-hanging fruit in our docs site 🍒. Although there is a good framework, we could both clean up and expand the site for commonly asked pointers.

We currently use docsify as a Github page for the docs, but that's amenable to change. Whatever we use should be automatic and 🤖.

Frequently asked docs

• Can I see some screenshots and videos?
• Which sites can I open right now?
• How do I run multiplayer?
• How do I plug in to my node.js app?
• Does it work with electron?
• Why doesn't https://youtube.com/ open?
• What is the "why" of the project?
• I have bug X.
• How do I run it in Magic Leap?
• How do I run it with Leap Motion?
• When is 1.0?

Chores

• Remove the huge cover page?
• <title/> tag?
• <meta/> tags?

How to fix these

Submit a PR to webmixedreality/webmr-docs 🥇

Does not start on Linux without GLIBC

Worker not catching script errors

window-worker is responsible for this functionality.

This can be repro'd with truncated scripts.

Make Exokit streamable

Streaming is a big thing. I spend more time on Twitch than I'd care to admit.

It would be good to harness that wave by adding stream integration into Exokit.

The integration should be developer-focused at first, since those are who we want streaming Exokit anyway. Maybe we can bundle OBS and make it a one-click setup?

Three.js examples sometimes only load in one eye of VR

https://threejs.org/examples/#webvr_vive only loads in left eye of Vive, assumed true for some other examples

Site test infrastructure

Background

Exokit essentially reimplements all of WebGL.

Problem

We have bugs and regressions with various sites as our code changes, which it does frequently! Some of these are soft-crashes in JS (throws), some is hardcrashes in the native bindings (SEGV, hangs), and some is incorrect rendering.

Solution

We could automate checking a list of sites against a build of Exokit in our CI environments on Appveyor and Travis CI.

In particular it seems it would be straightforward to check that some sites load and run to completion -- the harder part of this is differential rendering verification, which we might want to punt on till later.

One wrinkle is that we might want to make this work for all three OSes we support (Windows, OSX, Linux), so we might need three harnesses. Also, since Exokit is a graphical app that requires either OpenGL or an emulation of it (e.g. mesa on Linux), we'll need to make sure the CI environment runs such support.

This could be broken up into separate issues if this proves to be too much to chew at once 🍖.

Getting started

The three.js examples are a goldmine of simple but thorough test suites.

The ratio is high but not all of them examples load and render. However, those that do should continue to do so and can be whitelisted for verification.

Add progress bar to URL loader

So we an tell when things are loading/stuck.

WebXR support

https://webxr.io/

This one's a doozy.

But since Exokit is written in Javascript, we already support WebVR, and we have access to native OpenVR, and I envision a great deal of copypasta from our WebVR/OpenVR support.

I (@modulesio) will probably tackle this.

Add build instructions

Need to mention:

Requirements

• windows-build-tools
• MagicLeap building
• Node 9

Build methods

• npm i exokit
• Github Releases
• manual build

Releases unpack size is too large

[email protected] postinstall C:\Users\LucidWeb\Downloads\node_modules\exokit
> node downloader.js

[███.................] exokit-windows.zip v0.0.36 2778194 bps 16% 68.8sevents.js:183
     throw er; // Unhandled 'error' event
     ^

Error: invalid distance too far back
   at InflateRaw.zlibOnError (zlib.js:153:15)

Use OpenVR api for models

For responsive triggers/buttons and tooltips. Plus make it actually match the hardware.

Need to expose a web API for it.

libopenvr_api.so missing (linux)

Ran the install script with (X)Ubuntu 16.04 xenial x86_64 Linux 4.4.0-122-generic. Extracted the files just fine, when running exokit however some OpenVR library is missing. Is this expected? Didnt see any prerequisite requirements in the readme.

internal/modules/cjs/loader.js:707
  return process.dlopen(module, path.toNamespacedPath(filename));
                 ^
Error: libopenvr_api.so: cannot open shared object file: No such file or directory
    at Object.Module._extensions..node (internal/modules/cjs/loader.js:707:18)
    at Module.load (internal/modules/cjs/loader.js:589:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:528:12)
    at Function.Module._load (internal/modules/cjs/loader.js:520:3)
    at Module.require (internal/modules/cjs/loader.js:626:17)
    at require (internal/modules/cjs/helpers.js:20:18)
    at Object.<anonymous> (/usr/local/lib/exokit/native-bindings.js:2:18)
    at Module._compile (internal/modules/cjs/loader.js:678:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:689:10)
    at Module.load (internal/modules/cjs/loader.js:589:32)

Builds not reliable

A git submodule pull failed, the build succeeded anyway it seems for windows.

https://ci.appveyor.com/project/modulesio/exokit

Oculus DK1 support

https://developer.oculus.com/downloads/package/oculus-sdk-for-os-x/

I have an Oculus DK1 headset. It might be good to support that. I don't know whether DK2 is already supported.

Isolator crashing on windows

Missing node? Probably needs a full build download.

>exokit

C:\Users\Noah Schiffman\AppData\Roaming\npm\node_modules\exokit>IF EXIST ".\node\node.exe" (.\node_modules\isolator\lib\windows\isolator.exe -- .\node\node.exe .  )  ELSE (.\node_modules\isolator\lib\windows\isolator.exe -- node .  )
*os.PathError fork/exec .\node\node.exe: Access is denied.
C:/Users/Ultimus/go/src/github.com/modulesio/isolator/runner/winsandbox_windows.go:133 (0x8b9e12)
C:/Users/Ultimus/go/src/github.com/modulesio/isolator/cmd/runner/runner.go:156 (0x8bbd2d)
C:/Users/Ultimus/go/src/github.com/modulesio/isolator/cmd/runner/runner.go:36 (0x8bb592)
C:/Users/Ultimus/go/src/github.com/modulesio/isolator/main.go:236 (0x90d919)
C:/Users/Ultimus/go/src/github.com/modulesio/isolator/main.go:125 (0x90cf8a)
C:/Go/src/runtime/proc.go:198 (0x4306fe)
C:/Go/src/runtime/asm_amd64.s:2361 (0x45cbc1)

Add node 8 install warning

Since only node 9+ is supported.

Video load hangs crafting demo

Commenting out video support fixes it, but that's not a fix.

Youtube player

Would be nice if this worked:

exokit https://www.youtube.com/watch?v=dQw4w9WgXcQ

It could open the video in desktop mode, with enter VR/AR capability.

require("exokit") + exokit.load() does not work

Embedding Exokit in a nodejs project or running it directly from the node prompt does not work.

Code snippet:

Enter node and then paste

const exokit = require("exokit");
exokit.load('https://threejs.org/examples/#webgl_animation_cloth', {})

MacOS .app bundle is unsigned

Downloading from get.webmr.io yields the following error:

Magic Leap API + docs

We have supported the Magic Leap simulator for a while now. We will also support the real hardware when it ships!

However our approach/API exposition is not documented anywhere. We should clean up the API to conform to web expectations, with minimal deviation from WebXR, and then write up the documentation on https://docs.webmr.io 📖.

Fix aframe example

exokit.cmd . https://aframe.io/aframe/examples/animation/aframe-logo/?ui=false

Crashes.

Exokit landing page

Current site: https://exokit.webmr.io/

Needs work! We need to expand the site with:

• Download link for different OS's (currently https://get.webmr.io/ auto-selects; we should however have separate override links for Windows, macOS, and Linux ⛓
• Getting started guide 📚
• More pretty pictures and videos, maybe a gallery! 🖼
• Entry point to common things in the docs, such as downloading, features, contributing, troubleshooting, and so on 📖
• A succinct FAQ❓
• Entry points to socials we maintain (Twitch, Slack, Discord, Twitter, Youtube) 👩

Retina mac renders at quarter screen

MacBook Air is fine.

Battery info for controllers

I noticed in some VR game -- I think VRChat -- that you can see the battery level for each controller.

I wanted to add this feature to my own site, but it seems that the WebVR spec doesn't provide the info.

Is there a way battery level could be provided to sites anyway? What would that look like?

We can do something like:

if (navigator.userAgent.startsWith("MixedReality")) { .... }

But that's cumbersome and prone to breakage; probably the worst way to do it.

Loading non-MR site should throw error

Currently loading e.g. http://youtube.com/ will lock up the browser, but it should instead display an error in MR.

TypeError: Cannot read property 'responseStart' of undefined
    at http://youtube.com/:1:1807
    at http://youtube.com/:1:2575
    at _runJavascript (C:\Users\avaer\Documents\GitHub\exokit\core.js:3215:15)
    at HTMLScriptElement.on.innerHTML (C:\Users\avaer\Documents\GitHub\exokit\core.js:2486:9)
    at HTMLScriptElement.emit (events.js:182:13)
    at HTMLScriptElement.EventEmitter.emit (domain.js:442:20)
    at HTMLScriptElement._emit (C:\Users\avaer\Documents\GitHub\exokit\core.js:1343:40)
    at HTMLScriptElement.set innerHTML [as innerHTML] (C:\Users\avaer\Documents\GitHub\exokit\core.js:2528:10)
    at HTMLScriptElement.run (C:\Users\avaer\Documents\GitHub\exokit\core.js:2549:24)
    at element.traverseAsync (C:\Users\avaer\Documents\GitHub\exokit\core.js:3186:16)

CI: do not codesign PRs

Currently our builds will error during a PR build: https://travis-ci.org/webmixedreality/exokit/builds/377086347?utm_source=slack&utm_medium=notification

This is by design: the PRing Github account does not (and should not) have access to secure variables in the build, such as our code signing keys. But it would be nice for the build to not fail, since build success is highly useful info for PRs.

The straightforward fix for this would be to skip the code signing steps (for Windows on Appveyor and OSX on Travis CI if/when the PFX_KEY environment variable is not defined -- and maybe log the fact that this is happening for later debugging in case this behavior leaks into master.

CC @shawwn

Voice chat in home environment

It's almost great, just the ScriptProcessorNode implementation doesn't respect the buffer size, resulting in hitching across two different breeds of browser.

Fix for this is to use the current frame size (128) as a basis and queue up until we have enough data in LabSound to emit an onaudioprocess.

Related repository is the home environment itself: https://github.com/webmixedreality/exokit-home