servo/rust-webvr

UNMAINTAINED - WebVR API implementation for servo.

rust-webvr

Safe rust API that provides a way to interact with Virtual Reality headsets and integration with vendor specific SDKs like OpenVR, Oculus and GoogleVR (Daydream). The API is inspired on the easy to use WebVR API but adapted to Rust design patterns.

It's used in the WebVR Core implementation for Servo browser. This module can be tested outside of Servo and even be used on any vanilla Rust app.

Room Scale example:

Just run this command in examples/room folder

cargo run

Run room scale demo on android:

./run_android.sh

OpenVR tips:

In order to run with openvr on windows, openvr_api.dll must be available. Please make it either accessible in your path, or copy it into the examples/room folder.

Refer to The ValveSoftware openvr repository and head over to the releases section for more information.

rust-webvr's Issues

Getting eye parameters requires memory allocation

One of the webvr issues in Servo is that we freeze the eye parameters for a VRDisplay when the object is created, so if they change then the changes don't get seen by JS. servo/servo#22758

The obvious fix for this is to get the eye parameters from rust-webvr, but unfortunately the API for that is to get the VRDisplayData

rust-webvr/rust-webvr-api/src/vr_display.rs

Lines 12 to 13 in 4e30999

	/// Returns the current display data.
	fn data(&self) -> VRDisplayData;

and the display data includes the display_name as a String

rust-webvr/rust-webvr-api/src/vr_display_data.rs

Lines 7 to 15 in 4e30999

	pub struct VRDisplayData {
	pub display_id: u32,
	pub display_name: String,
	pub connected: bool,
	pub capabilities: VRDisplayCapabilities,
	pub stage_parameters: Option<VRStageParameters>,
	pub left_eye_parameters: VREyeParameters,
	pub right_eye_parameters: VREyeParameters,
	}

Since the name is a String, it has to be freshly allocated each time the display data is accessed, which is wasteful.

Possible fixes: separate out the eye parameters from the display data, or make the display_name a &'static str or Cow<'static, str> rather than a String.

Implement a test VRDisplay that renders to a GL window

Useful for testing VR code.

Controller connect/disconnect event

Right now, there are no connected/disconnected controller events. The events given by OpenVR are currently filtered to events from the HMD.

OpenVR also gives button press events.

https://w3c.github.io/gamepad/

https://github.com/ValveSoftware/openvr/wiki/VREvent_t

Should these events be provided in this library?

Why does `synced_frame_data` take a `&self` rather than a `&mut self`?

When servo uses a VRDisplay it has &mut access to it, so we could make synced_frame_data and friends take &mut self arguments rather than &self. This would remove some unnecessary uses of interior mutability.

Add TravisCI support

No support for 32-bit applications?

Tried to compile this for stable-i686-pc-windows-msvc, and got the following error:

error[E0512]: transmute called with types of different sizes
   --> C:\Users\Sgeo\.cargo\registry\src\github.com-1ecc6299db9ec823\rust-webvr-0.9.10\src\api\openvr\display.rs:152:46
    |
152 |         self.frame_texture.handle = unsafe { mem::transmute(layer.texture_id as u64) };
    |                                              ^^^^^^^^^^^^^^
    |
    = note: source type: u64 (64 bits)
    = note: target type: *mut std::os::raw::c_void (32 bits)

error: aborting due to previous error

For more information about this error, try `rustc --explain E0512`.
error: Could not compile `rust-webvr`.

I assume that this is because I'm compiling for 32-bit.

Room example fails to compile

   Compiling core-graphics v0.3.2                                                                                                                                                                                     
error: type `color_space::__CGColorSpace` is private                                                                                                                                                                  
  --> /Users/ajeffrey/.cargo/registry/src/github.com-1ecc6299db9ec823/core-graphics-0.3.2/src/context.rs:86:26                                                                                                        
   |                                                                                                                                                                                                                  
86 |             let result = CGBitmapContextCreate(ptr::null_mut(),                                                                                                                                                  
   |                          ^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                   

Looking at the output of cargo tree:

├── glutin v0.7.5 (https://github.com/MortimerGoro/glutin/?branch=android_fixes#ebbaaaf8)
...
│   ├── core-foundation v0.2.3 (*)
│   ├── core-graphics v0.3.2 (*)

So it looks like we should update glutin.

No support for desktop Oculus API?

Looking through the code, it looks like the Oculus SDK for Desktop isn't supported, only Oculus SDK for mobile (and Oculus SDK for Desktop indirectly through OpenVR).

Is this intended? Some Oculus Rift users would prefer applications to use the Oculus SDK directly if possible.

`VRServiceManager` isn't `Send`

Trying to share a ServiceManager between threads:

   |
81 |             .spawn(move || {
   |              ^^^^^ `std::cell::RefCell<(dyn rust_webvr_api::vr_display::VRDisplay + 'static)>` cannot be shared between threads safely
   |
   = help: the trait `std::marker::Sync` is not implemented for `std::cell::RefCell<(dyn rust_webvr_api::vr_display::VRDisplay + 'static)>`
   = note: required because of the requirements on the impl of `std::marker::Send` for `std::sync::Arc<std::cell::RefCell<(dyn rust_webvr_api::vr_display::VRDisplay + 'static)>>`
   = note: required because it appears within the type `(u32, std::sync::Arc<std::cell::RefCell<(dyn rust_webvr_api::vr_display::VRDisplay + 'static)>>)`
   = note: required because it appears within the type `std::marker::PhantomData<(u32, std::sync::Arc<std::cell::RefCell<(dyn rust_webvr_api::vr_display::VRDisplay + 'static)>>)>`
   = note: required because it appears within the type `std::collections::hash::table::RawTable<u32, std::sync::Arc<std::cell::RefCell<(dyn rust_webvr_api::vr_display::VRDisplay + 'static)>>>`
   = note: required because it appears within the type `std::collections::HashMap<u32, std::sync::Arc<std::cell::RefCell<(dyn rust_webvr_api::vr_display::VRDisplay + 'static)>>>`
   = note: required because it appears within the type `rust_webvr::vr_manager::VRServiceManager`
...

Since VRServiceManager is !Send, it can't be shared, even in a Arc<Mutex<VRServiceManager>>.

The VRDisplay is not provided with a Gl instance

When the VRDisplay is presenting, it does so (at least in Servo) in the context of a WebGL thread, that has access to an Rc<Gl> object, which provides GL bindings. This Rc<Gl> object is not provided to the VRDisplay, so if it wants to call any GL functions, it has to create its own GL bindings.

Release 0.2

This is needed to bump serde in Servo.

Support multiple displays in GoogleVR

rust-webvr/rust-webvr/src/api/googlevr/gamepad.rs

Line 43 in d1d5439

gvr::gvr_controller_state_update(controller_ctx, 0, gamepad.state);

Our gamepad code in GoogleVR only fetches the first controller. We should change it to fetch all controllers as the API supports it (and Lenovo Mirage needs it)

Deprecate some APIs

We should deprecate:

• sync_poses(), synced_frame_data() and presented_by_browser in favour of future_frame_data()
• render_layer() and submit_frame() in favour of submit_layer()

Then remove them in a breaking change.

We should have a way to get gamepad state asynchronously

Ideally, make it a variant of future_frame_data that gets you both the frame data and gamepads.

Currently we only have one async implementor of future_frame_data (GLWindow) and that has no gamepads, so it's not 100% necessary yet, but we may need it in the future.

cc @asajeffrey

VRViewport coordinates are ints or floats?

According to the docs, VRViewport coordinates are given as floats.

rust-webvr/rust-webvr-api/src/vr_framebuffer.rs

Lines 11 to 13 in 4e30999

	/// UVs defining the texture bounds to present to the eye in UV space: [x,y,w,h]
	/// Defaults to [0.0, 0.0, 0.5, 1.0]
	pub viewport: VRViewport,

According to the implementation, they're ints.

rust-webvr/rust-webvr-api/src/vr_framebuffer.rs

Lines 36 to 41 in 4e30999

	pub struct VRViewport {
	pub x: i32,
	pub y: i32,
	pub width: i32,
	pub height: i32,
	}

Move this repository to the Servo org

Hey @MortimerGoro, can you please move this to the Servo org, so that we install @bors-servo on it etc? Thanks.

rust-webvr shouldn't rely on features to conditionally export a single backend

Cargo features are supposed to be additive and are misused in this crate, for example the following command will fail on Android:

cargo build --features "googlevr oculusvr"

VRFrameData::from_bytes is unsound

This method builds a VRPose (for the pose field) out of a slice of bytes, but the VRPose struct contains Option<f64> types which don't have a specified layout and most importantly for which not all [u8] slices represent a valid value. VRPose::from_bytes should be unsafe.

https://github.com/MortimerGoro/rust-webvr/blob/cab969ee734908ebaac7667bbd4cd903e2135dd5/rust-webvr-api/src/vr_frame_data.rs#L54

OpenVRDisplay::new should be unsafe

OpenVRDisplay::new builds a new OpenVRDisplay out of raw pointers, and then does fancy unsafe things with these raw pointers, which means OpenVRDisplay::new itself should be unsafe.

https://github.com/MortimerGoro/rust-webvr/blob/3bef6ceee33c13ac6b7b6bef4c0eaae13eabf3ea/rust-webvr/src/api/openvr/display.rs#L38-L60

Any chance of a tutorial?

I'd love to write some VR in Rust. But this API is very technical (and OpenVR isn't any better), so a tutorial would be very, very welcome :)

Can we avoid blocking the WebVR render thread?

Currently, sync_poses is a blocking function, which blocks the WebVR render thread until the next animation frame. This means the render thread can't perform other useful work while waiting for the next animation frame. In Servo, the WebVR render thread is the same as the WebGL thread, so the WebGL thread is blocked, leading to issues such as servo/servo#22914, where webrender is blocked waiting on the WebGL thread.

Why is `register_vrexternal` safe?

There's a safe function VRServiceManager.::register_vrexternal (https://docs.rs/rust-webvr/0.10.0/rust_webvr/struct.VRServiceManager.html#method.register_vrexternal) which takes a VRExternalShmemPtr argumen, which can be constructed using the safe function VRExternalShmemPtr::new (https://docs.rs/rust-webvr/0.10.0/rust_webvr/api/struct.VRExternalShmemPtr.html#method.new).

Now currently this is all unimplemented!() but it's not obvious how this can be safe once the service manager does something with its VRExternalShmemPtr. Am I missing something?

Why is the `VRServiceCreator` API needed?

Why do we expose a VRServiceCreator API? Why does a user write:

FooVRServiceCreator::new()::new_service()

rather than just:

FooVRService::new()

Add android build to travisCI

This would allow building the room example as well as the Android-specific parts of rust-webvr-api.

Example doesn't work

pwd
$HOME/rust-webvr/examples/room

cargo run --example room

Updating registry `https://github.com/rust-lang/crates.io-index`
    Updating git repository `https://github.com/MortimerGoro/glutin/`           
error: failed to load source for a dependency on `rust-svr`                     

Caused by:
  Unable to update file:///$HOME/rust-svr

Caused by:
  failed to read `/$HOME/rust-svr/Cargo.toml`

Caused by:
  No such file or directory (os error 2)

Using openvr on windows leads to TrackedDevicePose_t uninitilized error

Hi,

i am using the HTC Vive on windows, with OpenVR.
If i start use the device like in the example, i get an error, TrackedDevicePose_t uninitilized.
The stacktrace leads to:
https://github.com/servo/rust-webvr/blob/master/rust-webvr/src/api/openvr/display.rs#L118

At first i tought i did setup the device wrong, but seeing the code two lines above. Can it be, that indeed struct should be initilized?

Rust Version:
nightly-x86_64-pc-windows-msvc (default)
rustc 1.53.0-nightly (07e0e2ec2 2021-03-24)

' panicked at 'attempted to leave type `api::openvr::binding::TrackedDevicePose_t` uninitialized, which is invalid', C:\Users\gojira\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib/rustlib/src/rust\library\core\src\mem\mod.rs:660:9
stack backtrace:
   0: std::panicking::begin_panic_handler
             at /rustc/07e0e2ec268c140e607e1ac7f49f145612d0f597\/library\std\src\panicking.rs:493
   1: core::panicking::panic_fmt
             at /rustc/07e0e2ec268c140e607e1ac7f49f145612d0f597\/library\core\src\panicking.rs:92
   2: core::panicking::panic
             at /rustc/07e0e2ec268c140e607e1ac7f49f145612d0f597\/library\core\src\panicking.rs:50
   3: core::mem::uninitialized
             at C:\Users\gojira\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\mem\mod.rs:660
   4: rust_webvr::api::openvr::display::{{impl}}::synced_frame_data
             at C:\Users\gojira\.cargo\registry\src\github.com-1ecc6299db9ec823\rust-webvr-0.19.0\src\api\openvr\display.rs:118
   5: shadow_of_truth::vr::VirtualReality::sync_data

VREyeParameters API incompatible with canted displays

I have a Pimax headset with canted displays. The matrices produced by GetEyeToHeadTransform thus contain a rotation. Not only does openvr/display.rs discard that data in fetch_eye_parameters, there's nowhere to put it in VREyeParameters. This will need to be fixed, and maybe not just for wide field of view headsets; the Valve Index also has canted optics.

JNIScope is unsound

JNIScope is a struct with only public fields (which means anyone is free to build new ones), but its Drop implementation dereferences raw pointers to call FFI functions on them. this is unsound.

https://github.com/MortimerGoro/rust-webvr/blob/cab969ee734908ebaac7667bbd4cd903e2135dd5/rust-webvr-api/src/jni_utils.rs#L9-L13

https://github.com/MortimerGoro/rust-webvr/blob/cab969ee734908ebaac7667bbd4cd903e2135dd5/rust-webvr-api/src/jni_utils.rs#L90-L98

Fill out mocking features in the mock API

We have a mock API, but there's no way to actually control it.

We should add it so that we can test things

Why is `VRDisplay` marked `Send` and `Sync`?

I understand why the VRService has to be Send and Sync, because it may be created on a different thread than its used (e.g. created on the main thread, used on the WebVR thread). I don't understand why a VRDisplay has to be. Why not query the VRService for its displays on one thread and then just leave it on that thread?

`start_present` and `stop_present` aren't called by Servo

Not sure if this is a servo issue or a rust-webvr issue, but the methods to start and stop presentation on a device are never called by servo. Shouldn't they be?

Use euclid for matrices?

At the moment the API uses raw float arrays to represent matrices, which means we can't use matrix operations (subtraction, multiplication, etc) and there is no record in the types of units.

Supporting devices that require code to run on the main thread

Currently the API has no concept of the "main thread" of an application, and assumes that VRServiceManager::new() and VRServiceManager::register_defaults() can be safely called from any thread (in servo, they're called from the WebVRThread). This has a mismatch with devices (such as MagicLeap) which have a main thread, and where the APIs to access the device capabilities aren't thread-safe.

Methods should return Result?

There are some methods that are currently required to be infallible, e.g. display.data() i https://docs.rs/rust-webvr/0.9.12/rust_webvr/vr_display/trait.VRDisplay.html, but the underlying device API may fail. Should the API be changed to return a Result?

Redesign the main API of this crate to be XR-focused

The WebVR API is deprecated, and I'm hitting roadblocks implementing stuff in WebXR since WebXR in many cases is more powerful than WebVR. Furthermore, it's tricky to work around this because most of the uses of this crate in Servo have nuanced soundness issues (#18) and I don't know how to avoid breaking those. For example, one of the issues I'm hitting is that WebXR expects gamepad information to stay up to date (unlike the Gamepad) API, and it's pretty hard to work that into the current model.

Some differences between WebVR and WebXR:

• WebXR supports inline immersive sessions, where there is no "display" but there is position and orientation information
• WebXR operates directly on pose matrices as opposed to view matrices
• WebXR treats gamepads as being attached to a session (display) and automatically kept up to date, WebVR thinks of gamepads as a wholly separate deal.
• The lifecycle model is somewhat different for the session

It might be worth finding an API that is a good middle ground between the two, as opposed to being primarily webvr-focused.

A big chunk of this crate seems unsound to me

Many APIs in there use a Arc<RefCell<T>>, and then use that as raw pointers somehow on the Servo side. That seems highly suspicious to me. @MortimerGoro Is there some documentation somewhere about how this is all used?