Comments (5)
Based on my experience with chrome's current implementation of this API, I believe it would be possible. You'd need to hold the camera texture and then draw it as an opaque quad at the back plane of clip space, which would effectively occlude the real time video frame underneath. You'd have to take care to make sure that the camera position you use when drawing is the same as the camera position from the time the texture was originally captured.
from raw-camera-access.
Thanks @nbutko, I suspected it would work something like that. My hope is that the held texture (which I assume could be a copy into a ring-buffer of textures or the like) can remain in webgl-driverland, rather than requiring a roundtrip readback out of webgl-driverland into JS-land and a subsequent writeback from js-land into webgl-driverland, as performance becomes more of a question with that type of arrangement. If someone could confirm there's a fast path for the texture copy, I'd be interested. The "opaque texture" concept still eludes me a bit, so I'm not sure if this would lead to any hindrances to a fast copy path.
One slightly unfortunate effect of using raw camera access for this usecase is that UA permission prompt will be required, even though access to the actual pixels isn't required... only delay of the output of those pixels is desired. If there were room in this API for delaying of camera output without actual access to the camera output, it could be quite useful, though I'm not sure if it's the best fit for this API? I do see that the Spec mentions caching here - https://immersive-web.github.io/raw-camera-access/#xr-web-gl-binding-section
Something like this caching mechanism combined with a manualPlayout(CachedUnreadableFrameHandle)
type function could allow for controlled delayed playout of an dev-unreadable cached camera frame texture, which would mean no additional UA permission prompts.
from raw-camera-access.
Based on my experience with chrome's current implementation of this API, I believe it would be possible. You'd need to hold the camera texture and then draw it as an opaque quad at the back plane of clip space, which would effectively occlude the real time video frame underneath. You'd have to take care to make sure that the camera position you use when drawing is the same as the camera position from the time the texture was originally captured.
Correct, this is how I'd imagine this would work, with a small clarification that an explicit texture copy is required since we (= WebXR) want to have full control over the texture lifetime. We could always hand out copies, but this would cause a performance hit for use cases that do not need the copy.
If someone could confirm there's a fast path for the texture copy, I'd be interested. The "opaque texture" concept still eludes me a bit, so I'm not sure if this would lead to any hindrances to a fast copy path.
I believe that as long as you issue appropriate WebGL commands from within the requestAnimationFrame callback, the copy should work fine. The "opaque texture" concept is there mainly to signal to the developers that they are not owning it so WebXR can (and, at least in Chrome, will) sweep the carpet from under them if the reference outlives the XRFrame.
I do see that the Spec mentions caching here - https://immersive-web.github.io/raw-camera-access/#xr-web-gl-binding-section
The caching that you refer to here is to ensure that we do not perform the same work over and over if the texture is requested for the same camera within the same rAFcb. The spec could've been silent about caching, in which case the UAs could still employ some caching, but I thought that explicitly calling out that parameter / state validation would be helpful to other implementers.
from raw-camera-access.
@bialpio Just a heads up for you, I opened a bug over on webkit - https://bugs.webkit.org/show_bug.cgi?id=229752
Obviously this is still just a draft, but I think it's good to raise visibility for all browser development even at this early stage. Thanks again.
from raw-camera-access.
(posted incorrect link in OP, post edited - https://bugs.webkit.org/show_bug.cgi?id=229752 )
from raw-camera-access.
Related Issues (9)
- Potentially resolving the privacy sensitivity issue HOT 3
- Notes from Face To Face 4/21/2022 HOT 1
- Interoperability and compatibility - web developers support HOT 5
- Explainers should lead with sample code, should not include IDL HOT 5
- Broken references in WebXR Raw Camera Access Module
- how would this work on non-video-mixed devices?
- How to get access to the pixels HOT 4
- When will this be shipped?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from raw-camera-access.