immersive-web/raw-camera-access

Spec draft: https://immersive-web.github.io/raw-camera-access/. Repository for experimentation around exposing raw camera access through WebXR Device API. Feature leads: Piotr Bialecki, Alex Turner, Nicholas Butko

raw-camera-access's Issues

When will this be shipped?

When will this feature be shipped? There is no way at the moment to use it on edege for hololense since it is only on chrome. Or am I wrong?

Notes from Face To Face 4/21/2022

We discussed the current shape of the API at the Immersive Web Working Group Face to Face.

There’s a growing recognition in the working group that a lack of camera access has held back adoption of WebXR compared to alternatives like getUserMedia. The ironic effect of having an artificially limited api is that it pushes people toward less privacy sensitive alternatives. The goal should be to have user consent that is potentially improved from existing flows (whatever that entails) but is powerful enough to empower the gamut of valuable developer use cases.

In order to make the current proposal fully compatible with headsets, the following changes might be needed

• frames are not coupled to XR frames, but come on a callback or some other mechanism
• frames are timestamped
• camera to viewer transform
• field of view for camera is provided

Here are some examples of reasons developers choose use alternatives to WebXR because this API is lacking:

• Custom Visual Effects: https://diamonds.samsmithworld.com/ renders diamonds with custom refraction of the camera feed.
• High quality reflections: https://www.8thwall.com/alivenow/freefire renders a metallic paint car with the users' environment reflected.
• Responsive Scale: https://www.8thwall.com/beemwebar/hearst-philips shows a hologram discussing body positivity that can be viewed comfortably at room scale or table scale.
• Photo Sharing: https://www.8thwall.com/aircards/dune-avatar enables photo sharing of a custom Dune avatar, with a Dune watermark embedded in the photo.
• Video Sharing: https://www.8thwall.com/alivenow/cadbury-fc-webar has users record and share a video with audio and embedded images painting their face with colors of their favorite soccer teams.
• Image Target Effects: https://www.8thwall.com/toolofnorthamerica/pizza-hut-pacman is a Pac Man game on top of a pizza box.
• Curved Image Target Effects: https://www.8thwall.com/tactic/white-castle animates comic on White Castle cups.
• Face Effects: https://www.jlab.com/pages/fitting-room lets users try on various audio accessories before purchasing.
• QR Code Scanning: https://8th.io/scan is a web assembly QR code scanner.
• Custom TensorFlow Models: https://www.campaignasia.com/article/heineken-asks-singapore-to-race-an-f1-car-in-ar/453689 detects when two Heineken bottles "cheers".
• Hand Effects: https://blog.tensorflow.org/2021/11/3D-handpose.html allows pages to detect hand position.
• Foot Effects: https://app2.vyking.io/vyking-web-ar/#/web lets users try on Air Jordans.
• Body Effects: https://www.auganix.org/geenee-announces-webar-full-body-tracking-sdk-for-3d-augmented-reality-try-on-and-avatars/ enables 3D WebAR try-on.

Interoperability and compatibility - web developers support

I'm preparing to kick off blink launch process for Raw Camera Access API, and one part of the Intent To Ship asks for signals from web developers - please take a look and let me know what you think! If my understanding is correct, we're looking for feedback along the lines of "the API [does / does not] solve my use case with [no / some / major] issues/workarounds needed", but don't feel limited by this formula!

So far the I'm aware of an issue around API ergonomics (see comment) - I think this can be tackled at a later stage, let's get the API out the door first and see what the main pain points are.

Pinging folks that may have some thoughts here: @nbutko, @tangobravo, @mrdoob, @elalish.

Explainers should lead with sample code, should not include IDL

Via the I2S thread, I started reading the explainer, and it seems like several best practices are missing and design directions are not enunciated:

• the end-user problem/benefit is not front-and-center
• the explainer includes proposed IDL, which should be relegated to a spec draft
• the explainer does not identify problems related to XR camera access that are not going to be addressed by the design (non-goals)
• the explainer is hand-wavey about why this is not being proposed as an extension to getUserMedia()
• why does the explainer not provide a way to also upload "raw" image data to a WebGL texture?
• for a "raw" image, the API seems to lack color space controls/hints/output, and does not document the format of the resulting texture. Why not?
• how does this feature handle multiple cameras? Stereo cameras? Why are views and cameras always linked 1:1?
• how will this work in the context of Offscreen Canvas?
• the considered alternatives section only contains a single other design, when we can easily imagine many different designs
• the spec doc does not link to the explainer
• the explainer does not link to the spec

Potentially resolving the privacy sensitivity issue

I understand there has been TAG feedback regarding the privacy concerns and how we can inform users of the risks.

Where it's difficult to explain to users the potential risks in giving a website unfettered access to their camera as they move around their environment pointing it at possessions and loved ones.

I would like to propose that WebXR features could be split in a way that is similar to position tracking's fine grain coarse grain approach. Where for us coarse grain are the privacy preserving approaches and fine grain are the ones that give more power to the developer at the risk of user's privacy.

This way we can encourage developers to not rely on the availability of the API because user's may choose not to allow it if they are in an inappropriate place and the developer should instead try to use a fallback behaviour to provide an approximate experience or an alternative experience.

Broken references in WebXR Raw Camera Access Module

While crawling WebXR Raw Camera Access Module, the following links to other specifications were detected as pointing to non-existing anchors, which should be fixed:

• https://www.w3.org/TR/webxr/#set-of-granted-features
• https://www.w3.org/TR/webxr/#requestanimationframe

_{This issue was detected and reported semi-automatically by Strudy based on data collected in webref.}

Can this allow for deferring of camera output on mobile devices?

Hi,

Thanks for putting this together, it's exciting. Can this indirectly also allow for deferring the output of particular camera frames? And if so, what would be the best way to use the API in order to achieve this goal?

There's a full thread about the usecase for deferring output of camera frames is here - immersive-web/webxr-ar-module#44

But I think @nbutko 's usecase explanation here is the most succinct I've seen:

Delayed drawing - on mobile phones an application might need to synchronize the vision with the camera feed.
-- Need to be able to hold an XR frame for ~2-10 animation frames before it is presented to the user with other results from vision.
-- from immersive-web/proposals#4 (comment)

(With the caveat is that we're doing remote rendering and need to sync that to the camera output, rather than syncing CV stuff. But the requirements are the same for both.)

How to get access to the pixels

Hello.

I just worked on my code that used the camera-access as it was implemented until Chrome 91.

Access to the camera works it seems, as I can run the sample code provided at the Chromium site successfully. But I fail to get the pixels out of the WebGLTexture object returned by glBinding.getCameraImage() call now.

As WebGLTexture is described as an opaque texture object, I assume that accessing the pixels is not meant to work. But as computer vision is listed as one of the use cases in the explainer, I have the impression access to the pixels is necessary somehow.

Would be interested what the situation and plan is around this subject.

how would this work on non-video-mixed devices?

Creating an issue to continue discussion from bialpio/webxr-raw-camera-access#1, the old repo is going to be read-only.