immersive-web/occlusion

occlusion's Issues

Should depth-sensing be merged into this?

All the uses i've seen of depth sensing have been to produce Occlusion effects.
Does focusing more on an occlusion API make more sense for now?

This could allow non-depth based methods to also be used.

/facetoface

Below-the-line approach to secure occlusion rendering

Hi all,

I work in Microsoft's Mixed Reality organization, looking at how to support occlusion for the web on devices like HoloLens using web frameworks like BabylonJS.

I've recently been looking into the diverse set of surface reconstruction methods that each mention their applicability for occlusion:

• ARKit
  • Occlude Virtual Content with a Mesh
  • ARMatteGenerator
  • ARDepthData
• ARCore
  • Depth API
• Windows Mixed Reality
  • Occlusion with Spatial Mapping
  • Depth buffer through Research Mode
• WebXR
  • depth-sensing
  • real-world-geometry

Overall, there seem to be two main themes: occlusion by geometry (reconstructed mesh) or by depth map (either from a SLAM-like algorithm or by LiDAR). Technically, no matter how we do it we're just rendering values to the depth buffer and doing z-testing.

Following the question posed in this thread, and given that surface reconstruction has improved significantly over the last year, I'm wondering if it makes sense to target a very simple API for doing occlusion that gives the significant added value of passively occluding virtual content without worrying about the implementation details. Basically, just let the browser decide how it wants to deal with assigning alpha in the color buffer or pre-filling the depth buffer, and don't expose any meshes or depth maps solely for the purpose of occlusion.

The idea behind hiding this information would be to avoid many of the privacy issues involved with providing real-world data, as well as allow different systems to implement occlusion as suits their hardware.

However, in order to have both security and performance we would likely need to make the depth buffer write-only. My impression is that we would want to render the relevant depth values for occlusion before handing the framebuffer to the app rendering the scene so that we could have perf wins by discarding fragments in the rasterization step, but that would require us to make sure the pre-filled depth buffer doesn't become a threat vector, as someone could read the depth values and map out the space in which the content was being rendered.

I'd just like to get the ball rolling and see what thoughts people have around these issues.

Adding some people I've seen in related conversations:
@bialpio @thetuvix @cabanier @toji @blairmacintyre @bricetebbs