Expose a hidden area mesh or a hidden area depth write function

[immersive-web]

Expose a hidden area mesh or a hidden area depth write function

[kearwood]

For Oculus (Desktop), this is exposed also, using ovr_GetFovStencil()

[kearwood]

For Oculus Go, it seems that enabling FFR (Fixed-Foveated-Rendering) would give most of this benefit.

[kearwood]

One more optimization...

The XRWebGLLayer should have a flag indicating that the submitted frame had the hidden areas masked with XRWebGLLayer.DrawHiddenMask. This can enable some optimizations by the platform...

Eg: A cheaper-to-integrate alternative to (LMS) lens-matched-shading and (MRS) mixed-resolution-shading can be achieved by having XRWebGLLayer.DrawHiddenMask() optionally fill a stipple pattern in the peripheral areas of the frame. When the submitted texture is blitted to the HMD's panel, the browser and/or VR compositor can then infer the colors of the omitted pixels in the stippled area.

This would give much of the benefits of MRS, foveated rendering, and the hidden area mesh mask, while only requiring two lines of code to be added to rendering engines such as three.js.

The browser could make an informed choice to offer the stipple-mask based variable shading rate depending on the particular platform attributes:

• Most mobile GPUs (eg, with TBDR) should inhibit shading of the hidden areas but not use the stipple effect. If FFR is enabled simultaneously, less (or none) of the depth buffer area should be cleared.
• For desktop GPUs or mobile GPUs without TBDR, the stipple-mask based variable shading rate should be optionally enabled by a flag on XRWebGLayer.
• For platforms that do not benefit (or implement) this optimization, they could simply implement XRWebGLLayer.DrawHiddenMask() as a null operation and ignore the flags added to XRWebGLLayer.

Engines such as three.js could safely call the function and add the flag without need to feature detect, resulting in no ill effects when the platform does not support it.

[kearwood]

Unreal engine supports a similar technique:

https://developer.oculus.com/documentation/unreal/latest/concepts/unreal-mbfr/

[kearwood]

I propose that we add to XRView's WebIDL:

readonly attribute FrozenArray<DOMPointReadOnly> visibleArea;
readonly attribute FrozenArray<DOMPointReadOnly> hiddenArea;

The DOMPointReadOnly values can be rendered with GL_TRIANGLE_STRIP. Any breaks in the triangle strip can be implemented by including degenerate triangles.
visibleArea and hiddenArea vertices are in screen space (0,0,0 = lower left, 1,1,0 = upper right)

Typically, the hiddenArea will cull 10-20% of the pixels for current VR displays.

This same 10-20% can be applied to the culling volumes, used by more advanced rendering engines -- It may be convenient at this time to also define culling volumes in the same format.

I'd suggest adding to XRView:

partial interface XRView {
  readonly attribute FrozenArray<DOMPointReadOnly> frustumCullingVolume;
  readonly attribute FrozenArray<DOMPointReadOnly> convexCullingVolume;
  readonly attribute FrozenArray<DOMPointReadOnly> concaveCullingVolume;
}

For frustumCullingVolume, convexCullingVolume, and concaveCullingVolume the vertices are in view space.

frustumCullingVolume is guaranteed to have 6 faces, with the near and far plane being parallel to one another.

convexCullingVolume is guaranteed to be a convex shape completely enclosing the volume visible in the XRView. convexCullingVolume may often be the equivalent of an extruded visibleArea mesh.

For frustumCullingVolume and convexCullingVolume, any point on the "outside" of any polygon in the strip is considered outside of the visible area.

concaveCullingVolume is not guaranteed to be a convex shape. More advanced testing of a volume intersection will be required.

Not all plaforms or UA's may be able to (or need to) expose separate culling volumes. It is acceptable for the same mesh to be returned for the "frustum", "convex", and "concave" volumes.

To support combined culling volumes, I'd suggest adding the same meshes to the XRViewerPose:

partial interface XRViewerPose {
  readonly attribute FrozenArray<DOMPointReadOnly> frustumCullingVolume;
  readonly attribute FrozenArray<DOMPointReadOnly> convexCullingVolume;
  readonly attribute FrozenArray<DOMPointReadOnly> concaveCullingVolume;
};

These meshes would be expected to be lazily generated by the UA on access but remain consistent for the duration of a frame.

[kearwood]

An alternate WebIDL could be more extensible via enums:

enum XRCullingMeshType {
  "visible-area",
  "hidden-area",
  "frustum-culling-volume",
  "convex-culling-volume",
  "concave-culling-volume"
}

partial interface XRView {
  FrozenArray<DOMPointReadOnly> getCullingMesh(XRCullingMeshType);
};

partial interface XRViewerPose {
  FrozenArray<DOMPointReadOnly> getCullingMesh(XRCullingMeshType);
};

We would need to define what "visible-area" and "hidden-area" meshes would mean when requested from an XRViewerPose, or to not allow this combination.

Also, perhaps it would be useful to return a "visible-area" and "hidden-area" mesh representing the combined XRView's rendered into a WebGLLayer.

[kearwood]

Simple engines could use the frustum culling area, while also adding some extrusion to account for latency in asynchronous streaming systems loading assets before they come into view.

An example of a more advanced rendering engine that could benefit most from the 10-20% reduction in the convex and concave culling volumes would be a "clustered forward rendering" engine, used commonly in modern VR applications.

The "clustered forward rendering" algorithm involves sub-dividing the typical 6-sided frustum volume into smaller frustum-shaped parts on all 3 axis. It then performs culling of objects and light volumes on each subdivided part independently. Only the lights and objects contributing to the output of each subdivided part are included as each of these parts are rendered separately.

Bulk culling can be performed at this cluster level, prior to performing the sub-culling operations and lighting calculations for each of the cluster parts. The cost of computing intersection of these clusters against a more complex (convex or concave) culling volume is more than offset by the savings in setting up and rendering the entire cluster that is culled.

[klausw]

Edit: >180 degree FOV is not an issue for per-view culling since each view would stay below that. I think there's still an issue that apps may want an overall FOV, but overall I think that encouraging apps to do per-view culling and reminding them to be careful about merging culling volumes seems like the right approach.

We're starting to get HMDs with >180 degree overall FOV, and it's mathematically impossible to express that as a frustum. A more general culling volume could handle this, as could other approaches such as specifying combined FOV angles instead of top/right/bottom/left planes, but even in that case it's likely that applications may make bad assumptions and not handle this correctly, for example by trying to convert >180 degree angles to a frustum. I'm not sure what the best way is to solve this, but we should clarify in the spec that this is a possibility.

Separately, I think that a hidden area mesh approach could potentially be useful for AR occlusion if it can fill in arbitrary depth data as opposed to just a boolean visible/hidden state. That's a separate topic, but if we do want to go in that direction it would be nice to reuse data structures and mechanisms if appropriate.

[kearwood]

In the case of HTC Vive, this is expected to result in a fill-rate reduction of 18%.

[kearwood]

Some other platforms don't provide the mesh directly, but rather provide a function to populate the depth values. Either we would expose both methods using two functions, or just expose the depth value rendering function and implement it within the browser using the meshes returned by other platforms.

This method would be even easier for engines to implement. Effectively, they would just call this function immediately after clearing their render target, then perform the other rendering passes as normal.

We would not want to enable this by default, as it could interfere with post-fx processes such as depth of field and screen space reflections. Such effects are rarely used in VR content; however, so this would most often be a win to enable.

[kearwood]

I'd be tempted to expose such a depth-buffer-mask function directly into XRWebGLLayer, as XRWebGLLayer will have all the needed information to inform the shape of the mask:

XRWebGLLayer.DrawHiddenMask()

As some engines may have multiple render passes, this function should not assume which render target to write to. It could perhaps write to the currently active target, as per the WebGL render state.

[kearwood]

Possible WebIDL:

partial interface XRWebGLLayer : XRLayer {
readonly attribute unsigend long foveatedMaskLevel;
void clearHiddenAreaDepth();
}

[kearwood]

If XRWebGLLayer.foveatedMaskLevel is non-zero, WebGL content is not expected to perform reconstruction of the stipped-out pixels.

This would require some exploration of how the FOV levels of the stipple region(s) are selected for various values of foveatedMaskLevel.

[kearwood]

If a WebGL based engine includes a post-fx pass, it may be desirable to perform the reconstruction within the same shader. This might suggest a need to separately enable the stippling by clearHiddenAreaDepth() and the implicit reconstruction by the browser/platform.

[avaer]

Does this issue encapsulate more generic depth buffer population for things like AR occlusion, or is the focus on rendering optimizations?

[klausw]

(Edited my previous comment, I had missed that this culling is per-view, and each view will be <180 degrees.)

[bjornbytes]

Note: the XR_KHR_visibility_mask extension exposes its visibility mask in a triangle list instead of the triangle strip topology proposed here, so an engine aiming to target both OpenXR and WebXR would need to do a manual conversion for one of the APIs if it wanted to expose the mask in a consistent format (this would at least impact LÖVR).

It may also be good to specify the vertex winding.

[toji]

Last week during TPAC we reviewed all of the issues in this repo.

As mentioned above OpenXR does support this feature via an extension, so it's still a feature that's seeing some support today. Discussion on the call, however, indicated that fixed foveated rendering was likely to provide some of the same benefits while being both an easier knob for developers to turn and being a bit friendlier to reprojection-heavy environments (which WebXR content tends to be.) Fortunately this is already a planned feature in the Layers API.

Areas where the visibility mask may still be uniquely useful are headsets like the Varjo which use inset displays. The visibility mask could allow for the overlapping area to be stenciled out and save a little bit of performance.

Given this, the group doesn't have immediate plans to expose this API but we'll leave this issue open for reassessment in the future.