Should plane data be available outside of rAF?

[immersive-web]

Should plane data be available outside of rAF?

[bialpio]

Revisiting this.

I think the right way to go here is to expose a plane space relative to which the vertices are expressed, such that all the vertices have an Y coordinate set to 0.0. In addition, the plane space's Y axis should align with the plane normal, and the origin of the coordinate space defined by that plane space should lie at a plane's center (both ARKit and ARCore seem to define plane center as the center of the bounding rectangle). The plane data is guaranteed to be valid within an XRFrame that is active (so in rAF + events), but will be accessible outside of it as well.

To justify those choices:

• Exposing vertices relative to plane space allows the application to easily extend the plane to infinity.
• If the plane only shifts its position without changing its boundary, the polygon does not need to be updated (so vertex buffers can be reused as long as the shape remains the same).
• Placing the plane space at the plane's center provides some guarantees about where the anchor will be created to the application. There's still an issue with bounds drastically changing from frame to frame (where should the anchor be created in that case?), but immersive-web/anchors#37 & immersive-web/anchors#42 should account for that.

I'm working on a draft spec text that includes those choices, let me know if the above makes sense.

[johnpallett]

From #2, Blair says: Why are things only valid in the rAF? As your example shows, the first thing people will do is copy it. This seems massively wasteful for non-trivial amounts of world info. Why not just say it's valid through the start of the next rAF?

[johnpallett]

From #2, bialpio says: Good question. The primary assumption with this design is that the geometry will be used for drawing, and drawing is only allowed in rAF callback. We discussed this internally and while some use cases (e.g. using planes for occlusion, or rendering planes) are explicitly related to rendering, even non-rendering use cases (e.g. scanning a room) are likely to provide a visual indicator of state, which effectively makes them rendering use cases.

We also recalled some of the conversations around hit testing which considered linking geometry results to the camera frame (on visual SLAM systems), and there were concerns at that time that the geometry could too easily be out of sync with the rendering and could cause jittering or other artifacts.

If the app still needs to use the information after rAF, it can do so by copying the data. My examples were only doing so for demonstration purposes - just to have variables that can be passed in to other functions from rAF callback. I’ll modify them to make it more clear.

[johnpallett]

From #2, Blair says:

Good question. The primary assumption with this design is that the geometry will be used for drawing, and drawing is only allowed in rAF callback.

It will also be used for physics, for example. And there are already demonstrations of using workers for physics; there have been demonstrations of drawing in workers, but Spectre/Meltdown put a stop to that (by forcing vendors to eliminate the necessary shared memory, I seem to recall).

And when people what to do analysis of the geometry (e.g., to detect planes or other objects), that will be very amenable to being done in workers.

We discussed this internally and while some use cases (e.g. using planes for occlusion, or rendering planes) are explicitly related to rendering, even non-rendering use cases (e.g. scanning a room) are likely to provide a visual indicator of state, which effectively makes them rendering use cases.

Rendering is the obvious use, I agree. But limiting data use arbitrarily just because of this seems like a poor choice. Is there are good reason to do it, aside from this?

We also recalled some of the conversations around hit testing which considered linking geometry results to the camera frame (on visual SLAM systems), and there were concerns at that time that the geometry could too easily be out of sync with the rendering and could cause jittering or other artifacts.

I agree that we absolutely need to know the current state of the geometry in the rAF. I was questioning the "not valid outside the rAF" point.

If the app still needs to use the information after rAF, it can do so by copying the data. My examples were only doing so for demonstration purposes - just to have variables that can be passed in to other functions from rAF callback. I’ll modify them to make it more clear.

But, my question still stands: is there a reason to require this? Yes, if I am passing data to a worker, I very likely will copy it, but I'd actually prefer to do this in the worker, not in the main thread, for performance reasons.

What we actually need is the opposite: a way to pin the object until you are done with it, and then release it so it can be reused internally.

These data structures may get very big. As I walk around my house with a Hololens on, or ML1, the size of the world geometry continues to grow, and future devices will generate finer and finer meshes. We cannot design and API that requires this data be copied each frame for non-trivial use cases.

[bialpio]

Key observation:
User Agent needs to have a window of time to update the objects if we decide on the approach of updating the objects that we’ve handed out (as currently stated in the explainer). We also need to somehow express the period of data validity to the application.

In order to solve this while still allowing the objects to be used by the application outside of rAF callback, we’d have to implement some kind of object pinning, as you’ve mentioned. This will mean that subsequent rAF callback will not be executed until the application unpins all of the objects (this slightly worries me as now we’re introducing manual resource management - it’s error-prone). To me, it effectively means that the application could have just taken more time in its rAF callback - am I missing something here?

I think that our current API design doesn’t rule out adding this in the future, so for now I’m tempted to stick to limiting access to the objects only to rAF callback. Added benefit is that it’s fairly easy to express this - “Data for frame N can be accessed only in rAF callback for frame N, if you need it later, you need to copy it.”.

[cabanier]

I don't quite understand why the data should only be available during raf time.
Shouldn't the coordinates all be relative to a reference point that doesn't change?

[bialpio]

I don't quite understand why the data should only be available during raf time.

Well, technically we could make it available outside of rAF callback as well, I tried to touch upon that in the response above. We’d have to guarantee some period where the plane data is stable (and conversely, some period where UA is allowed to update the plane data). I think we could relax the “rAF only” requirement and say that the data is valid during rAF and other callbacks (e.g. XRSession.onselect event handler), or even further, as long as we communicate clearly when things will not change (e.g. “plane data is valid from the time the first rAF callback for given frame was invoked until just prior to when the first rAF callback is invoked for subsequent frame”). What do you think?

Shouldn't the coordinates all be relative to a reference point that doesn't change?

There are multiple options possible:

1. Plane vertices are relative to a fixed space (reference point). The space may move relative to other spaces as the system’s understanding of the world changes, but the vertices remain fixed relative to it (assuming that that no new data about the plane's polygon is available).
2. Plane vertices are relative to the center of the convex hull. Changing / expanding the polygon also causes changes to the reference point, since it's the center. The reference point may also move due to changes in system’s world understanding.
3. All plane’s vertices are expressed in a single reference space. The plane’s vertices are all updated as the system’s understanding of that plane’s position relative to the world changes. Plane pose is no longer used to express translation, just the rotation is needed.

To look at a couple of implementations, in ARCore, the plane polygon’s vertices are all relative to the center of the convex hull of the plane (link) so the reference point might move from frame to frame (option 2). In ARKit, the extents are also relative to the center, but if I read the docs right (link), the polygon is relative to a reference point that shouldn’t change (option 1). We can choose not to specify the exact behavior of the plane’s pose (i.e. don’t say it’s the center of the convex hull, just say that polygon vertices are relative to it, and maybe that the pose lies on the plane, to keep all Y coordinates of vertices 0) - it would allow us to accommodate both ARCore’s approach and ARKit’s approach.

Any thoughts on how other libraries might expose this data would be welcome!

[cabanier]

I'm unsure how ARCore or ARKit work but it seems that a plane should always be relative to a reference space. Why would the position of a plane be different from an input controller?

Any thoughts on how other libraries might expose this data would be welcome!

Magic Leap returns plane data in world position. This means that we could return the plane data directly using the XRBoundedReferenceSpace

[bialpio]

I'm unsure how ARCore or ARKit work but it seems that a plane should always be relative to a reference space. Why would the position of a plane be different from an input controller?

Right now, ARCore and my proposal for planes work as described in option 2: the plane center is an XRSpace & you can query its pose via XRFrame.getPose(to_space, from_space). The plane's XRSpace can get updated potentially every rAF - to me this means that the plane center's position behaves exactly like an input controller's XRInputSource.targetRaySpace. The plane vertices are expressed relative to plane's space. ARKit seems to follow something closer to option 1.

[cabanier]

I have a better understanding of XRSpaces now so your proposal sounds good.