Gamepads in inline mode

[NathanaelA]

Gamepads in inline mode

[Manishearth]

I believe the specification does not prevent browsers from supporting this; this is an issue for implementors.

[NathanaelA]

@Manishearth - The specification doesn't currently allow gamepads in inline mode based on this section:
https://www.w3.org/TR/webxr/#xr-device-concept

The default inline XR device MUST NOT report any pose information, and MUST NOT report XR input sources or events other than those created by pointer events.

This is a proposal to relax the restrictions for "inline" mode only in the event the developer requests actual controllers. But also trigger a permissions check like "interactive-*" modes to to preserve the privacy which is the reason why inline doesn't currently support it.

[Manishearth]

The specification supports accessing the gamepads directly via navigator.getGamepads().

As for the ability to get input sources: Input sources are only useful if you have reference spaces to compare them against, and no current XR system supports inline with floor/local/etc spaces.

I would be open to relaxing that constraint when the feature is requested. Though I'd want signals from implementers that they're interested in doing this.

[NathanaelA]

The specification supports accessing the gamepads directly via navigator.getGamepads().

So even in "inline" mode, this would be allowed by the spec, because my understanding is because they (the existing controllers) are used as "pointers" in "inline" mode they can't really be used as standard gamepads, since they basically act like a mouse pointer effectively. I assume this is why both inline and requestfullscreen don't activate/trigger the navigator.getGamepads code paths. The idea here, is using "requiredFeature": ["gamepad"] for inline mode basically says I'm fully charge of the controllers and I need all the input, so browser your done with them...

As for the ability to get input sources: Input sources are only useful if you have reference spaces to compare them against, and no current XR system supports inline with floor/local/etc spaces.

Not necessarily, did you see what I am using the game inputs for? It is a flat 2d canvas, I don't need anything other than the inputs. I do eventually want access to the pointing data and where a click occurs because of games like X-Com or point-and-click type games like Kings Quest, Return to Zork. That code is something I'll probably work on next week to my vr framework code, but for a large number of games where a mice aren't used, the plain gamepad data is all I need.

I would be open to relaxing that constraint when the feature is requested.

A little confused, isn't that what this is, a feature request to allow relaxing the constraint only if the developer needs it?

Please note, this decision impacts me very little now, I've already learned enough webgl to be dangerous 😀 and built my own mini xr-framework to deal with all the issues that I ran into, but I am requesting this to make the way a lot smoother for all the other non-gl web developers to get their 2d content into xr worlds very easily. Gamepad data is critical in a many cases, and right now the road blocks are massive to get it and it could be, imho, trivially simplified.

[Manishearth]

I don't need anything other than the inputs

What do you mean by "the inputs"? That's what I'm trying to get at here.

A little confused, isn't that what this is, a feature request to allow relaxing the constraint only if the developer needs it?

No, I am saying that I am open to relaxing the current constraint that code cannot request the gamepads feature in an inline session. When I say "feature is requested" I'm talking about WebXR features, not this feature request.

[NathanaelA]

@Manishearth - So in my case I'm using the Oculus/Meta Quest 1&2. (Will probably support others headsets later, but only have a Q1/2, so can only test that it works with them at this point).

What do you mean by "the inputs"? That's what I'm trying to get at here.

The inputs being Buttons 0-6, and the Axis controls on both controllers. I don't need the default pointer/click mouse emulation for the browser, I literally need the controllers exactly as I get them in "immersive-vr" mode.

I need to use all the inputs to remap to controls in the game. If you have an Oculus, you can check my site and you can see now 3 games now work in VR as I finished my remapping code and so I can bring games in much easier. In the first couple games they are simple side scrollers and so both controllers basically act the same, meaning you only have to use one controller to play the game. Axis controlls movement, buttons control jumping/firing/etc.

However, with Magic Carpet for instance (still working on making the controls feel good, which is why it isn't listed as vr yet), The left controller's axis is used to point/turn the carpet, the right controller axis one is mapped for moving forward/backwards, and strafing side to side. The trigger buttons are the mapped the identically, but A/B/X/Y buttons all do something different. I basically need the vr controllers as-is just like I get them in "immersive-vr" mode, with the gamepad object that I can query for the buttons/axis's. I also will eventually need the pose data as I believe that is needed to figure out where the pointer is at, because some games like Return to Zork/Kings quest are mouse games where you point and click, and I'd like to make the point controller work as a mouse controller effectively...

Does that help you understand?

[Manishearth]

The inputs being Buttons 0-6, and the Axis controls on both controllers

These are all exposed through the Gamepad API directly, yes?

It sounds to me like the problem is that the profile mappings aren't always the same (implementors are free to do this). The profile string exposed by XRInputSource does seem useful here.

I think it would be reasonable to allow the gamepad feature be requested, permissioned, in inline contexts. I would like to have signals from implementors that they want to allow this, though.

[NathanaelA]

These are all exposed through the Gamepad API directly, yes?

In immersive-vr (and I assume immersive-ar) mode they are exposed as part of the gamepad object off of the XRInputSource objects. In inline mode the gamepad object is null.

I assume they cannot be exposed currently as gamepads in inline mode, because how would the browser know that the developer wants it to switch from default behavior (i.e. mouse pointers) to gamepad mode. If it started guessing and guessed wrong, it would break the default mouse pointer behavior. At this point all inline mode is basically mouse mode content, so I don't think we want to change the defaults nor have it "guess".

This feature request is to allow me the web developer to tell the WebXR system (& browser specifically) that I do not want the default browser behavior in inline mode. I actually DO want them exposed as the XRInputSource.gamepads, and I'm willing to take full control of the inputs just like I would in immersive-vr mode by passing a requiredFeature flag to the initialization system.

It sounds to me like the problem is that the profile mappings aren't always the same (implementors are free to do this). The profile string exposed by XRInputSource does seem useful here.

Not really a profile issue (afaik), the issue is the defaults is the current expected behavior, changing them would break any existing inline content, which we don't want. I think the proper way to do this is to add requiredFeature: gamepad then I as the developer am telling the WebXR system to take full control of them the same way I do in immersive-vr

I think it would be reasonable to allow the gamepad feature be requested, permissioned, in inline contexts.

👍

[Manishearth]

In immersive-vr (and I assume immersive-ar) mode they are exposed as part of the gamepad object off of the XRInputSource objects. In inline mode the gamepad object is null.

I'm talking about navigator.getGamepads() which exists independantly of the WebXR API. I'm trying to understand why those gamepads are not useful, because in inline mode that's what you're supposed to use to get button info.

(and we've already had discussions at the spec level that that is intended behavior: XR input sources can be Regular Gamepads too)

[NathanaelA]

I appreciate the time you are taking to understand the issue, hopefully this will finally clarify it. 😀

I'm talking about navigator.getGamepads() which exists independantly of the WebXR API. I'm trying to understand why those gamepads are not useful, because in inline mode that's what you're supposed to use to get button info.

Oh, I'm very aware of that API, I use it else where. That was where I first started my bunny trail into this whole mess and ended up having to make my own very minimalist XR framework just to get access to the gamepad data. If you see my opening comments on this issue; I mention I tried using that API.

Here is the breakdown for the current Oculus browser on a Quest 1 & 2:

Mode	API	Gamepad Supported
Browser	Just listen for gamepadconnected event	❌
Browser	listen for gamepadconnected event and requestFullscreen()	❌
WebXR	inline session and listen for gamepadconnected event	❌
WebXR	inline session and use XRInputSource.gamepad	❌
WebXR	Interactive-VR session and use XRInputSource.gamepad	☑️
Proposed Feature
WebXR	Inline session w/ RequireFeature: gamepads	🥺 🙏

I assume the gamepad axis and buttons are NOT exposed in anything outside of a WebXR interactive-vr session, because the controllers are already being used by DEFAULT for the browser navigation, clicking, selecting, back button, etc. Afaik, there is no way to let the browser know that I want the controller for any other uses, so afaik, you can't get it exposed outside of the interactive-vr session. That is the reason behind my idea, it seems the most feasible is to allow me the developer to request them without changing defaults, and breaking anything in an inline session which is the closest thing to a normal 2d browser session.

[Manishearth]

Okay, I see. Yeah, that is not something the spec can fix, that's an implementor concern (see immersive-web/webxr-gamepads-module#19 for some prior discussion), and it's something we want to leave up to the implementor. From the point of view of the spec, navigator.getGamepads() is where this functionality can be exposed if needed.

If we do relax the requirement on requesting gamepad in inline sessions you'll have the same problem; we can allow implementors to accept that feature string, but we cannot require they support it.

[NathanaelA]

If we do relax the requirement on requesting gamepad in inline sessions you'll have the same problem; we can allow implementors to accept that feature string, but we cannot require they support it.

Yep, I realize that -- the spec is optional, but this at least gives a place for this to actually be codified as a possibility, just like layers option or even the immersive-vr are not mandatory. Right now it is impossible. If it is an option in the spec, it means it is can be implemented and is a future possibility. 😀

[Manishearth]

What I'm trying to say is that it is already a possibility via the gamepads API.

(In general I'm reluctant to add something to the spec unless implementors give a signal they are interested in supporting it because otherwise we have no basis on which to spec it)

[NathanaelA]

What I'm trying to say is that it is already a possibility via the gamepads API.

Ok, lets talk specifics. Because one of us is confused as to the how the navigator.getGamepads() API would or even could work... I'm going to use an Oculus Quest because that is what I've experimented with, and I assume since it is currently the most prevalent standalone VR device.

Terms

First lets define some terms, to keep things easier to discuss (at least for me) since the terms have been somewhat used interchangeably up to now.

• Gamepad = Corded or Bluetooth type Gamepad device like a Logitech F710 or XBox controller.
• XRController = Oculus Touch type controller / device.
• Gamepad API = W3C standard Gamepad API [https://w3c.github.io/gamepad]
• XRGamepad API = Immersive-Web XR spec [https://github.com/immersive-web/webxr-gamepads-module]
• Browser = Oculus Browser

Objective

What is the flow or potential flow for a browser to allow the XRController to be used as if it is a pure gamepad where all buttons and all axis's are reported to either of the gamepad api's and then the XRController's has no default "browser" actions and the XRController's are no longer emulating a mouse on a standard web site.

Current Behavior

Unless you activate an XR immersive-vr session, the Browser uses the XRController's to act like a mouse with a few extra features like (backward navigation, page movement, etc). Both XRController's act identically as far as I can tell.

Issues

1. The Gamepad API requires a user based signal from the gamepad to trigger both the gamepadconnected event and for the actual gamepad structure to be filled out for getGamepads() under the Gamepad API.

   • What specific behavior on a XRController's would be considered the user facing signal? If the user tries to click on a button/link on the page using the primary trigger and hits (or misses it), what makes the browser think this is a user based signal to activate an gamepad api mode? As far as I can tell, there is no actual valid "user" facing signals from the XRControllers to actually signal the users intention to switch modes into a Gamepad api mode because the buttons are already used for default click & browser navigation. Since their also is currently no API calls in either spec to switch modes, you can't wire up a button to switch modes as the user action.

   • The Gamepad spec does work great for a actual Gamepad and it is a valid signal for an actual Gamepad to be used as a Gamepad because the Gamepad is NOT used for anything else, so clicking any button on a Gamepad triggers the gamepad API system. This spec also works great if you are hooking an actual Gamepad device to an XR device, because again the Gamepad is not used for anything until it is activated via the user signal.

2. On computers & phones/tablets the mouse, keyboard, and touch screen all continue to work by default on a page when gamepad devices are active. So default navigation is unimpeded. With XR devices, the XRcontroller's are ALREADY in use as the browser "mouse" and their is no backup (i.e. a no mouse/keyboard/touch screen) to use for any possible navigation. So switching the device to an Gamepad API mode had better be 100% certain, otherwise doing this accidentally will greatly confuse the user when the controller seems dead for all purposes.

   • What is the developer API signal stating that I as a developer, for this specific page, I need the XRController to be used as gamepads and I no longer need the XRController as a browser mouse?
   • What differentiates it to the browser from a developer wanting only a normal Gamepad verses using an XRController as an Gamepad? Taking over the XRController breaks default navigation and the developer needs to take that into account; as many api's have user action requirements to use so they need to be triggered by an user interaction BEFORE the developer takes over the XRcontroller.

3. Afaik, this group has no control over making any changes to the traditional W3C Gamepad Api spec. So if any changes need to be done to support a feature in any spec relating to an XR controller, it has to be done only to the XRGamepad spec which this body controls.

   • This means changes to the traditional gamepad api are out of scope, please feel free to correct me if I'm wrong.
   • We are actually talking about XR controllers, so it seems to me the WebXR spec's would be where it would be best handled.

Summary

With the current Gamepad API, there does NOT seem to be a good way for either the User or the Developer to signal intent that not only meets the rules to activate the device, but also to make sure we don't break any existing sites. We can't assume that me tying to the gamepadconnected event (or using getGamepads) is for a XRController rather than for a standard Gamepad device. Making that assumption would break sites since that code is frequently ran during initialization of a site which is also running on non XR devices and/or maybe the developer only wants an actual gamepad devices, not any XRControllers. There also is no Gamepad API call that allows you to request a subset of devices (i.e. navigator.getGamepads(["standard","xr-standard"]); to give the Gamepad api a hint as to what is allowed or requested, so you can't subvert XRController's default behavior by assumption when you use the getGamepads call.

So technically under the current existing api's, there appears to be NO good way for this class of WebXR device to be used as a general gamepad on a normal 2d site using any of the existing api calls.

Proposed Solution(s)

1. A gamepad API getGamepads optional parameter to specify the developer wants actual xr-standard devices. I don't believe this group can do it, but it is a valid solution, as it would mean the developer is signalling to the browser they are aware that they willing to hande a xr-standard device via the standard Gamepad API, they know they need to make sure to trigger it after the user has interacted with the page to eliminate user action based api permission restrictions.

2. XR inline Session, with an optionalFeature or requiredFeature of gamepad -- now the XRController could be mapped to a Gamepad API getGamepads, or a XRInputSource.gamepad (where the gamepad data lives really doesn't matter for this discussion), but the developer has sent the Browser/Api a solid signal of asking for the XRController to act like a gamepad as a feature of the inline session. This signal keeps from breaking any sites and specifies that the developer actually wants to take over control of the XRControllers fully and since it has to be triggered by a user action it also fulfills the same behavior requirement as many other user action based api restrictions to allow those API's to be used also before disconnecting the default XRController behavior.

3. A new navigator.xr.requestSession( "gamepad") mode that does nothing except activate the XRControllers as a gamepad as a normal gamepads API disabling the XRController mode. This also would allow a developer to signal his intent and would be only a WebXR spec change, and doesn't require the "inline" mode to be changed and allows inline to be depreciated as some other issues have discussed. Using XRSession.end( ) would revert the XRcontrollers back to XRControllers.

[Manishearth]

Ah, I think I understand now.

There's an implicit assumption in your post that the reason that implementations do not currently expose XR gamepads via getGamepads() is because of the potential ambiguity with someone interacting with the page. While I do think this ambiguity exists, I'm very much not convinced that that is actually the reason (cc @cabanier ?). In general we have seen less interest in implementors exposing XR-like features to 2D sessions (see #756) and I imagine it's a result of that.

If the user tries to click on a button/link on the page using the primary trigger and hits (or misses it), what makes the browser think this is a user based signal to activate an gamepad api mode?

I was assuming these would only be accessible via getGamepads() and would not trigger gamepadconnected but I also think it is completely valid and in the spirit of the spec to trigger them on first interaction.

So if any changes need to be done to support a feature in any spec relating to an XR controller, it has to be done only to the XRGamepad spec which this body controls.

No, it can be done upstream too, we have done this in the past. It's just a matter of where things are supposed to belong. Advice on how to handle XR gamepad controllers using navigator.getGamepads() really does live upstream.

A gamepad API getGamepads optional parameter to specify the developer wants actual xr-standard devices. I don't believe this group can do it, but it is a valid solution,

I don't think this is necessary or that it solves the problem, since getGamepads() already can return xr-standard devices and you can just test that.

I think what you are actually looking for is a way to hint to the UA that "no, really, please include XR controllers in the gamepads API even if they're being used for regular input too". Which does solve the ambiguity problem, though as I mentioned before it's unclear if that's the main blocker for implementors supporting this.

developer actually wants to take over control of the XRControllers fully

This bit is unlikely to ever happen; I do not think we will ever let inline sessions take full control of XRControllers. We can let them accept button presses but it would security-wise be rather bad to let them completely take over.

XR inline Session, with an optionalFeature or requiredFeature of gamepad

Yeah, this is my preferred solution as mentioned before; however I am still reluctant to go down this path without a signal from implementors

/agenda Talk about gamepads and also local/viewer spaces in inline sessions

[cabanier]

Ah, I think I understand now.

There's an implicit assumption in your post that the reason that implementations do not currently expose XR gamepads via getGamepads() is because of the potential ambiguity with someone interacting with the page. While I do think this ambiguity exists, I'm very much not convinced that that is actually the reason (cc @cabanier ?). In general we have seen less interest in implementors exposing XR-like features to 2D sessions (see #756) and I imagine it's a result of that.

In the case of the non-webxr experience, the Quest browser doesn't have access to the controllers. Instead, the system service that draws the environment is the one that gets their input.
If we were to somehow forward the button state to the 2D browser, we would have to be very careful that this only happens when the browser window without the chrome is the one that the user is interacting with. Also, would third party iframes work?

[Manishearth]

So I think the WebXR-side design for this is:

• We allow inline sessions to request the "gamepad" feature
• Input sources obtained this way have locked or null pose
• We talk about the capture problem: inline sessions should not be able to capture controller inputs used outside the session
• We normatively require UAs handle the capture problem
• We can suggest solutions:
  • Only inputs within the webpage are captured
  • Do something similar to the capture API where focusing on the canvas will hide controllers, grey out browser chrome, and require trusted UI interaction to exit)
  • Only allowed in fullscreen modes

However, we are getting into tricky design questions and we still don't have UAs who wish to expose this, and I'm wary of doing the above without an actual implementor committing to a mitigation strategy: I don't want to build castles in the sky. So my preference is to potentially hone this design but not add it to the spec until an implementor registers interest.