benvanik/vr.js

NPAPI plugin to expose fun VR devices to Javascript.

vr.js

An experimental NPAPI plugin for Chrome and Firefox that exposes fun VR devices.

vr.js, in conjunction with a required native browser plugin, exposes the Oculus Rift and Razer Hydra to Javascript in a performant, easy-to-use way. The library makes it simple to query the device values in just a few lines of code but also handles more advanced things like computing all the math required for rendering lens distorted scenes. If you want, it even has a slick API for easily rendering the distored scene that should be easy to drop into any WebGL application. There's also an example three.js wrapper under examples/js/effects/ that works pretty well.

Oh, and though it's possible to use node and WebSockets to get the sensor data I don't recommend it - the latency is simply too high (~10ms). This plugin allows for a latency similar to if you were developing a native application against the Oculus SDK and, when running on a correctly configured computer, will be pretty darn good.

NOTE: the Oculus SDK doesn't like sharing devices - you must close other Oculus apps before using this in your browser and must close your browser if you want to run another Oculus app. Lame :(

If you want to see something really crazy, check out the experimental pure Javascript driver for Chrome. Pure Javascript device drivers, pretty insane, huh?!

Supported Devices

• Razer Hydra
• Oculus Rift Development Kit

Supported Platforms

• Windows

  • Chrome 26+ (surprisingly good performance)
  • Firefox 20

• Mac OS X

  • Chrome 26+
  • Firefox 20
  • Safari 6

If using the experimental Chrome USB driver you can run on both OS X and Linux.

Installing

• Download the repository ZIP
• Extract to some path
• Mac OS X:
  • Run bin/install.sh to install the plugin.
• Windows:
  • Chrome:
    • Open Chrome to chrome://extensions
    • Check 'Developer mode' and click 'Load unpacked extension'
    • Select the bin\ folder in the path you extracted the ZIP into
  • Firefox:
    • Open an administrator command prompt
    • cd to bin\ in the path you extracted the ZIP into
    • Run install.bat
    • You should see a successful message box

Demos

You must have the plugin installed before they will run:

• Raw Data
  • Source: examples/raw_data.html
• Sixense Sensor Data Visualization
  • Source: examples/sixense_sensor_viz.html
• Rift Sensor Data Visualization
  • Source: examples/rift_sensor_viz.html
• Simple WebGL Demo
  • Source: examples/rift_cube_demo.js
• Three.js Floating Boxes Demo
  • Source: examples/threejs_boxes_demo.html

Documentation

Code is heavily commented - it's best to read that. Everything is in the vr.js file right now.

If you want fancy HTML docs, see the Online Documentation.

Tips

Launch Chrome with Flags

There are many features in Chrome you can disable to decrease latency. Install a Canary or Chromium build and launch it as follows:

/Applications/Google\ Chrome\ Canary.app/Contents/MacOS/Google\ Chrome\ Canary --disable-gpu-vsync --disable-image-transport-surface --disable-threaded-compositing --in-process-gpu

Some things may not work quite right, but it'll be faster!

Calibration

In the future there will likely be a simple calibration tool added to the JS library, but for now the best way to get your IPD is to run Team Fortress 2's calibration tool and copy the value out.

OS X: Disable Vsync

Download Apple's 'Graphics Tools for Xcode' from their download page and run the 'Quartz Debug' app. Disable 'Beam Sync' at the bottom. This may cause some tearing but reduces latency.

Windows: Disable Aero

This removes a frame of latency.

• Right click on desktop
• Personalize
• Choose Windows 7 Basic or Windows Classic

Don't Overlap the WebGL Canvas

If you place any other DOM element on top of the Canvas rendering your content you may cause extra browser compositing that can slow down your rendering. Since you have to draw your entire HUD/etc distored anyway, avoid placing any UI on top of the Canvas or adding any CSS effects to it (rounded corners, etc).

Write Fast Code

Use requestAnimationFrame for your rendering and always render as fast as possible.

Future Ideas

3D DOM

Using CSS transform and matrix3d it'd be possible to position any DOM content correctly. Then, once CSS Shaders are available in browsers, the DOM content could be distorted/color corrected.

Pure Javascript Drivers

An experimental pure Javascript driver for Chrome is available here. It works on OS X and Linux and requires a small tweak before it can work, but shows the promise!

Chrome apps have the chrome.usb API allowing direct access to devices. Implementing the sensor communication and sensor fusion code in Javascript allows apps to work on any OS Chrome runs on (ChromeOS too!) without the need for plugins. Unfortunately it's restricted to packaged apps only, not the general web, and the API does not support HID devices.

Mozilla is also considering a USB API, WebUSB, however it seems to have stalled.

Building

Windows

Visual Studio 2010 or 2012 is required for building on Windows. The Express editions should work, just make sure to get the VC++ 2012 for Desktop variant. Other dependencies are included in the repo.

Uninstall any previous installation of the npvr DLL before continuing.

Check out the git repo and generate the Visual Studio projects:

git clone https://github.com/benvanik/vr.js.git
cd vr.js
git submodule init
git submodule update
make-gyp.bat

Open build\npvr\vs2010\npvr.sln and build. The outputs will be placed into build\npvr\Debug\.

Run the following to prepare the debug version and register it with Firefox:

make-debug.bat

To use in Chrome:

• Open Chrome to chrome://extensions
• Check 'Developer mode' and click 'Load unpacked extension'
• Select the build\npvr\debug\ folder

Debugging

Make sure to uninstall the pre-built binary and instead install the plugin from the Debug build directory. When trying to rebuild the plugin always ensure the browsers that have loaded it are closed.

Chrome

Install the Chrome Canary. Running it on its own (instead of your main Chrome instance) makes it much easier to debug/restart/etc.

Exit all previous instances and launch from a command prompt:

"%LOCALAPPDATA%\Local\Google\Chrome SxS\Application\chrome.exe" --debug-plugin-loading --enable-logging --v=1 --plugin-startup-dialog

Open a page with the plugin and wait for the popup telling you the process ID. Switch to Visual Studio, go Debug -> Attach to Process, select the Google plugin process with the matching process ID, and attach.

Firefox

Try to run Firefox with no other pages loaded.

Disable the out-of-process plugins to make things easier: open about:config, find dom.ipc.plugins.enabled, and set it to false. Restart Firefox.

Launch, open the page, and attach to firefox.exe in Visual Studio.

License

Apache 2.0, except the np_* code.

Some portions of the code are from the official Oculus SDK and fall under their license. I hope that's cool :)

Credits

A lot of the code comes from the official Oculus SDK. Some math snippets from Brandon Jones's gl-matrix.

vr.js's Issues

Results On OSX 10.8.5

the command

bin/install.sh 

doesn't work.

I manually copy the plugin to my plugin directory and it works perfectly in Safari 6.1.2.
The plugin crashes in Chrome Version 33.0.1750.152 and crashes in Firefox 27.0.1.

Embed tag causes scroll bars

I'm trying to run fullscreen apps using vr.js. The embed tag that is used (presumably to load the plugin) seems to be impossible to completely hide. This leaves unsightly scrollbars in my app 😢

display: none seems to prevent it from loading, as does position: absolute. Any ideas?

.getOrientation()?

Hey,

I'm trying to track down the point where the Oculus rotation values are taken from the USB and brought into the vr.state.hmd.rotation variable. So far I've been able to track it to line 489, but I have a few questions.

1. what is global['vr_driver']? Right now, I'm assuming 'global' is the window object and ['vr_driver'] is some property on it, but I'm not to sure. When is 'global' defined? Is it the window object or am I completely off on that?

2. The .getOrientation() method that takes 'state.hmd.rotation' as a parameter...I have no idea what it means. Is it a primitive function? I can't find any reference to it in vr.js. Is it defined elsewhere?

Thanks for anything you can tell me. Just trying learn. The code I reference is below. Awesome library btw.

vr.DriverDataSource.prototype.poll = function(state) { //line 485
var present = this.driver_.isPresent();
state.hmd.present = present;
if (present) {
this.driver_.getOrientation(state.hmd.rotation); //<<---THIS LINE line 489
} else {
state.hmd.rotation[0] = state.hmd.rotation[1] = state.hmd.rotation[2] = 0;
state.hmd.rotation[3] = 0;
}

file './experimental/usb-driver/demo/demo.html' is missing

Hi,

The file '/home/jamie/Downloads/vr.js-master/experimental/usb-driver/background.js' attempts to launch 'demo/demo.html' when the experimental usb driver plugin is launched. Unfortunately, the file doesn't exist.

Maybe it got missed out somewhere or not committed. But it means at present you can't use the experimental usb driver.

Build an OSX plugin.

Should be easy enough with gyp. Need to figure out the NPAPI differences on OSX. May need a 64bit version? (I believe both Chrome and FF run in 32 on all OSX variants today)

Keyboard Movement Does Not Follow Direction You Are Facing

I've noticed in the "Three.js Floating Boxes Demo" that your perspective follows the oculus rift, but movement with the keyboard does not. So, for instance, when you first open the demo if you don't move your head, UP on the keyboard will move you forward, but if you rotate your head to the left or right, UP on the keyboard still moves the same direction even though you are facing a different direction. This should work similar to the Three.js Pointer Lock Demo http://threejs.org/examples/#misc_controls_pointerlock where UP on the keyboard will move you in the direction you are facing.

Render target in StereoRenderer is too wide.

It's currently sized to the entire display size, when really it only gets one eye rendered into it at a time. This is a waste of memory. Either write the warp shader to operate on the entire render target and render both eyes into it before compositing, or halve the size of the RT.

Detect locked OVR devices.

If an application is already open and using the HMDDevice it will be enumerable and creatable but only yield a 0,0,0,1 quat from the sensor data. This is likely detectable, and it'd be nice to return better state errors so that apps can tell the user ('close other apps/etc').

DK2 Support

Any plans to support DK2 (SDK 0.4)?

Use chrome.hid instead of chrome.usb.

As of Chrome 38 there is a chrome.hid API which provides access to devices that are claimed by the system HID driver.

Build a Firefox package for auto install.

Add a mock mode so things work with no devices.

Chrome doesn't get OVR info.

FF can get it, but Chrome never does. Rotation comes back, but the device info query always returns the empty string.

Flesh out the Chrome plugin and upload to CWS.

Find the alternative to NPAPI

How do you think...

http://blog.chromium.org/2013/09/saying-goodbye-to-our-old-friend-npapi.html

Mutltipass rendering has clearing issues.

I have a project that calls render 3 times per frame and composites the result. It basically renders 3 individual scenes on top of each other. In plain ol' Three.js it looks like this:

(Pardon the coffee script)

# setup renderer
@renderer = new THREE.WebGLRenderer antialias: yes
@renderer.setSize window.innerWidth, window.innerHeight
@renderer.autoClear = no # do not autoclear on each call to render

# Render a frame
render: ->
  @renderer.clear yes, yes, yes
  @renderer.render background.scene, @camera

  @renderer.clear no, yes, yes
  @renderer.render midground.scene,  @camera

  @renderer.clear no, yes, yes
  @renderer.render foreground.scene, @camera

At the start of each render, we clear all frame buffers (color, data, stencil), then we render the background. Now we clear only the depth and stencil buffers and then render the midground. And again for the foreground. We've now cleared the last frame and composited the next frame from three render calls.

Transitioning this strategy to use the OculusRiftEffect was tricky. The problem is here:

// from the end of THREE.OculusRiftEffect.prototype.render
this.renderer_.render(scene, this.eyeCamera_, undefined, true);

That last argument is the forceClear argument. And it's hardcoded to true. This means my foreground render was clearing the previous drawn scenes.

In order to fix this issue, I had to hack the library. I'm not saying this is the solution, but it worked for me.

// Added 4th `clear` argument
THREE.OculusRiftEffect.prototype.render = function(scene, camera, vrstate, clear) {
  // ... other code ...
  this.renderer_.render(scene, this.eyeCamera_, undefined, clear);
  // ...
}

And now my own render loop can now look like this:

# Render a blank scene and clear the buffer
@oculusRenderer.render emptyScene, @camera, undefined, true

# With a cleared canvas, render each layer
@oculusRenderer.render background.scene, @camera, undefined, false
@oculusRenderer.render midground.scene,  @camera, undefined, false
@oculusRenderer.render foreground.scene, @camera, undefined, false

Rendering a blank scene to control clearing is totally messy, but it seemed the only way I could figure out to get a clean clear when I wanted it. So I'm not sure what the proper API is for this case, or how to implement it, but there is clearly an issue here that should be addressed with multipass rendering.

Plug in does not work with HD prototype

I have no problem using the regular devkit, but the plug-in seems unable to detect the hd prototype. I have tried Chrome, Safari, and Firefox on a mac, and it was unable to detect it in any of them.

The HD prototype does work with the demo included in the Oculus SDK.

Plugin not working under Mac OS as of Chrome Version 39

As stated in http://www.chromium.org/developers/npapi-deprecation, Chrome does not support 32bit NPAPI plugins anymore.

"On Mac, version 39 onward, Chrome will be 64 bit only and this will imply 32 bit NPAPI plugins will stop to work on Chrome on Mac and there will be no way to whitelist it by policy. 64 bit plugins however can still be whitelisted and will continue to work until overall NPAPI removal in September 2015."

Could you please push a MacOS version built for 64bit architecture?
Thanks! Felix

Support Razer Hydra on Mac

Sixense has a Mac SDK at http://sixense.com/macsdkdownload, so it should be possible to get the Hydra working with the NPAPI plugin on Mac.

Plugin occasionally fails to detect correct resolution

It seems like the plugin sometimes detects the incorrect device resolution, which makes applications using it display a canvas that is far too large.

This could somehow be a "getting stuck" issue.

Rift Not Detected

I am getting an error with it not recognizing the rift.
The plugin is enabled and the rift can see my screen, however the program is not recognizing the rift

usbhid detach on linux doesn't work

On ubuntu 12.04 adding the 40-oculus.rules file, unplugging the device, restarting the udev service and plugging the device in does not prevent te usbhid driver to latch onto the device.

uname -a: Linux wintermute 3.2.0-41-generic-pae #66-Ubuntu SMP Thu Apr 25 03:50:20 UTC 2013 i686 i686 i386 GNU/Linux

usb-devices section for the oculus rift:

T: Bus=08 Lev=01 Prnt=01 Port=00 Cnt=01 Dev#= 6 Spd=12 MxCh= 0
D: Ver= 2.00 Cls=00(>ifc ) Sub=00 Prot=00 MxPS=64 #Cfgs= 1
P: Vendor=2833 ProdID=0001 Rev=00.16
S: Manufacturer=Oculus VR, Inc.
S: Product=Tracker DK
S: SerialNumber=8D8723894855
C: #Ifs= 1 Cfg#= 1 Atr=a0 MxPwr=100mA
I: If#= 0 Alt= 0 #EPs= 1 Cls=03(HID ) Sub=00 Prot=00 Driver=usbhid

Plugin sometimes gets "stuck" in detected or not detected state

Where it will claim to be detecting a rift that is not plugged in or vice versa. This will sometimes continue across Chrome restarts, forcing a re-install of the plugin (after which it seems to work again)

Feature request: support for accelerometers/phones (for Google Cardboard)

NPAPI plugin broken in Windows 8.1

npvr.dll is unable to load because of missing dependencies.
Dependency Walker lists the following DLLs as missing:

API-MS-WIN-CORE-KERNEL32-PRIVATE-L1-1-1.DLL
API-MS-WIN-CORE-PRIVATEPROFILE-L1-1-1.DLL
API-MS-WIN-SERVICE-PRIVATE-L1-1-1.DLL
API-MS-WIN-CORE-SHUTDOWN-L1-1-1.DLL
EXT-MS-WIN-NTUSER-UICONTEXT-EXT-L1-1-0.DLL
IESHIMS.DLL

This seems to be a recent but common issue in Windows 8.1 (see http://answers.microsoft.com/en-us/windows/forum/windows8_1-performance/32-bit-application-fails-to-start-after-81-upgrade/b825723e-e2a2-4c8f-bd1f-10446a5d7059).

I am currently trying to rebuild the .DLL using Visual Studio 2012 under Windows 8.1. Will report back.

Update to oculus rift sdk 0.2.4

PIpeline interruptTransfer requests.

It's possible to pipeline interruptTransfer requests to ensure there's no lag in sampling. This may help to increase the sampling rate and reduce latency, at the cost of potentially more CPU usage.

Publish to npm

I see that you have a package.json already set, it'd be useful if you published lib/vr.js to npm in order to make it easier to integrate with.

Cleanup OVR shutdown.

The call to OVR::System::Destroy(); in ovr_manager is currently commented out as it causes a hang. The problem seems to be that the Thread::FinishAllThreads() call deadlocks waiting for threads that are never properly cleaned up by the SDK (or Firefox kills them before this code executes).

It'd be nice to clean this up so that shutdown was clean in case the SDK starts requiring it.

Support plug and play of HMDs.

Currently the HMD must be plugged in by the time the plugin loads for it to be detected. It'd be nice to periodically query for it, either by having that done via the JS library or a detection thread that runs in the background.

The only way to check is by polling, as the messages from the device manager do not allow reentrancy.

Improved udev rule?

I've now managed to get the vr.js experimental usb driver working on linux. I stumbled upon an article which clarifies the difference between the PROGRAM and RUN assignments in udev rules:

http://www.reactivated.net/writing_udev_rules.html#external-run

I think it suggests that RUN would be a better assignment than PROGRAM for the udev rule contained in 'experimental/usb-driver/hacks/40-oculus.rules'. So the new rule would be:

SUBSYSTEM=="usb", ATTR{idVendor}=="2833", ATTRS{idProduct}=="0001", MODE="0664", GROUP="plugdev", RUN+="/bin/sh -c 'echo -n $id:1.0 >/sys/bus/usb/drivers/usbhid/unbind'"

I've checked this and it still works. It also might be worth pointing out that a user may not have the "plugdev" group on their system (depends on distro). This just needs to be any group that the user who'll be running the usb-driver belongs to.