immersive-web/lighting-estimation

A proposal for adding lighting estimation API to WebXR Device API. Feature lead: Kip Gilbert

lighting-estimation's Issues

Consider renaming "Image Based Lighting" section to "Cube Map Textures"

As per the feedback by @elalish on the initial PR, we should consider renaming the "Image Based Lighting" section to "Cube Map Textures".

"Image based lighting" implies not only the presence of cube map textures but also how they should be interpreted by a renderer. That said, the next section is labeled "Spherical Harmonics", which by itself does not imply how they should be interpreted by the renderer. At the least, we should have a glossary of these terms. Perhaps this could be discussed in the CG call to get consensus.

Add support for an optional stabilization plane

Can we add another variable stabilization_depth to XRRenderState?

It defines a distance where the engine will focus its stabilization. Even though it's possible to estimate it from the depth buffer, it's really only the author that really knows this information.

We could specify that non-AR sessions ignore this value.
A reasonable default would be 1.0f and the value must be between depthNear and depthFar.

Promises should not be returned by a function called on XRFrame

If we can consider the requesting of xr-global-light-estimation or xr-global-reflection features at the XRSession as a request to capture and generate the lighting estimation data, it should not be necessary to use promises in the functions called on XRFrame.

Restrict usage to one tab at a time?

If 2 browser tabs are able to use this API at the same time, I'm concerned about the potential for cross-site tracking. Would it be possible to restrict usage to only 1 tab at a time?

Determine internal format of the Reflection Cube Map

With ARCore on Android the internal format of the reflection cube maps returned by the API is arrays of GL_RGBA16F values that the app is then expected to upload to a texture. On Apple's ARKit the internal format is listed as bgra8Unorm_srgb and already delivered in texture form.

How should this disparity be handled by the API? Largely WebGL can sample from either texture type without knowing the internals, but I can imagine it would make a difference in how developers would want to process the values in their shaders. Additionally, some formats (GL_RGBA16F) won't be available with WebGL 1 without several extensions enabled. Given that, it seems like we have a few options for how to approach this:

1. Allow each platform to return it's native format without alteration, probably with an attribute to advertise what the internal format is and make developers just deal with it. If the format isn't supported by the current context just fail. Lowest impact for performance but hardest to deal with for developers.
2. Allow each platform to return it's native format unless the format isn't supported by the current context, in which case convert it to something that is supported.
3. Force all platforms to report the same format, in which case one platform is likely to get screwed in terms of performance and possibly numeric accuracy.
4. Support multiple formats and allow the developer to request their desired one, indicating which format is ideal for the given platform. Would allow developers to avoid conversions if they want to do the work, but would probably require ALL platforms to implement internal conversions for at least once case.

Describe temporal filtering required to avoid fingerprinting

Rapid changes to incoming light can provide information about a user's surroundings that can lead to fingerprinting and side-channel attacks on user privacy.

As an example, a light switch can be flipped in a room, causing the lighting estimation of two users in the same room to simultaneously change. If the precise time of the change can be observed by the sites, it can be inferred that the two users are co-located in the same physical space.

Another example occurs when a nearby monitor is playing a video, such as an advertisement. The light from the display reflects off many surfaces in the room, contributing to the observed ambient light estimate. A timeline of light intensity changes can uniquely identify the video that is playing, even if the monitor is not in direct line-of-sight to the XR device sensors.

The lighting-estimation specification should require temporal filtering of the light estimation to avoid such attacks. A low-pass filter effect can be achieved by averaging the values over the last several seconds. For single scalar values representing light intensity or color, such as XRLightProbe.indirectIrradiance this can be applied directly. SH's have a convenient property that they can be summed and interpolated by simply interpolating their coefficients, assuming their orientation is not changing.

After temporal filtering, the values should be quantized. Any vectors, such as XRLightProbe.primaryLightDirection should be quantized in 3d space and always return a unit vector.

The lighting estimation should describe the needed quantization and temporal filtering for all members in the webidl.

Current device support for lighting estimation in `immersive-ar`?

Apologies that this is not spec-related, but it is tangential and I wasn't sure where else to ask: Does anyone know of a headset that supports this in AR mode currently? My Android phone does, but Quest 3 doesn't work in my own personal tests, or using https://threejs.org/examples/#webxr_ar_lighting - It warns "Feature 'light-estimation' is not supported for mode: immersive-ar" and then throws "Session does not support feature light-estimation" when calling session.requestLightProbe (Quest Browser v32, Chromium 122)

@cabanier - any ETA on Quest 3 supporting this?

Consider un-bounding the number of coefficients for XRLightProbe.sphericalHarmonicsCoefficients.

The number of SH coefficients, as returned by XRLightProbe.sphericalHarmonicsCoefficients should perhaps be allowed to be increased at the discretion of the UA and AR platform. Increasing the number of coefficients would allow more distinct light sources and more accurate shapes for such lights without needing to use an HDR cubemap texture.

If a render engine does not need the extra coefficients, they could simply truncate the list returned by the UA. In the event that an engine is expecting more coefficients than the UA returns, it can substitute "0" for the additional coefficients. This is possible due to the nature of SH, without introducing aberration in the rendered image.

It should be noted that if a very large number of SH coefficients are provided by the UA, the resulting detail will converge towards that of a cube map texture. UA's should treat SH with a high number of coefficients as privileged information affecting the privacy of the user.

Consider exposing the XRReflectionProbe off the XRSession object

As the XRReflectionProbe's cube map can likely be used for multiple frames and is not updated every frame, perhaps it should be exposed on the XRSession object rather than XRFrame. It should also have an event to indicate when a new cube map is available.

The WebGL rendering engine could then interpolate between the last and new cube map when a new one is available.

Consider removing `XRLightProbe.sphericalHarmonicsOrientation` and `XRReflectionProbe.orientation`

The orientation of the platform specific API's lighting estimation may not match the XRSpace orientation. XRLightProbe.sphericalHarmonicsOrientation and XRReflectionProbe.orientation were included to avoid the cost of generating rotated versions of SH and cube maps.

Rather than storing the relative orientation, perhaps the UA should generate rotated variants aligned with the XRSpace of the viewer for global lighting estimation.

Lighting estimation for WebXR

There's already an API on the web for Lighting Estimation (https://w3c.github.io/ambient-light/) but it currently only supports getting the ambient intensity.
It would be useful for improving the realism of AR rendering to have slightly more information than that. For example the color temperature would be helpful and is generally available on the underlying XR platforms that would back WebXR. There are also other more advanced lighting options available on some platforms that would be worth investigating as optional features. One example is this from ARKit: https://developer.apple.com/documentation/arkit/arenvironmentprobeanchor

Need to formally define the integration of irradiance in XRLightProbe's sphericalHarmonicsCoefficients and its relationship with the discrete samples accessed by XRReflectionProbe

XRReflectionProbe's cube map should be described as representing discrete samples of incoming light, integrated only over the area represented by the pixel at each mip level.

XRLightProbe's sphericalHarmonicsCoefficients represent the samples that would be sampled by a fully diffuse material. The integration for the SH would be defined as covering a semi-sphere centered around the incoming light vector. For such a wide integration, 9 SH coefficients are able to reach 1% accuracy for any particular vector.

It is possible to accurately generate XRLightProbe SH coefficients from an XRReflectionProbe's cube map though integration of semi-sphere's; however, the reverse conversion can only represent a low frequency detail approximation.

Consider adding ability to generate a low resolution HDR cube map from SH

Based on feedback from @elalish in the initial PR:

If we don't get permissions, can we still return a low-res version created from the output of XRLightProbe? That way we don't force anyone to implement spherical harmonics and directional lights if they just do IBL.

I would like to hear feedback from other UA vendors to see if they would also consider implementing a convenience function for generating low resolution HDR cube maps from the SH coefficients.

As there is computation involved and some renderers may be optimized for using SH coefficients directly, these HDR cube maps should not be calculated unless requested. Perhaps this could also be subscription based.

Broken references in WebXR Lighting Estimation API Level 1

While crawling WebXR Lighting Estimation API Level 1, the following links to other specifications were detected as pointing to non-existing anchors, which should be fixed:

• https://www.w3.org/TR/webxr/#xrsession-list-of-enabled-features
• https://www.w3.org/TR/webxrlayers-1/#session

_{This issue was detected and reported semi-automatically by Strudy based on data collected in webref.}

Describe method of quantization to reduce fingerprinting

Method of quantization of values should be described. For inspiration, we should be informed by the Sensor API spec for best practices.

Consider forcing use of session feature flag to enable this API

Lighting estimation has non-trivial privacy and performance concerns, and many apps may not need to use it. Depending on the underlying implementation, there may be a non-trivial startup and per-frame cost for having this API available even if no calls are made. Given that WebXR already has a system for optional/required session features, it may be beneficial for performance, battery life, and privacy, to introduce a new feature flag to allow apps to signal their intent to use this API.

Refactor spec to be subscription based

Feedback from the W3C F2F meeting and feedback on the initial PR is supporting the pattern of "subscribing" to values that change from frame to frame, rather than resolving a promise each frame. We should refactor XRFrame.getGlobalLightEstimate() and XRFrame.getGlobalReflectionProbe() to be subscription based. This should be cohesive with the subscription pattern proposed for XR hit testing.

Use light probes with PRT SH functions to express ambient lighting

I would like to suggest that the lighting estimation be represented as light probes enumerated on an XRFrame.
A modern method of expressing ambient lighting is through Pre-computed Radiance Transform (PRT), encoded as a Spherical Harmonic (SH) function. For more detail: https://patapom.com/blog/SHPortal/
Possible webidl to express such a light probe:

[SecureContext, Exposed=Window] interface XRLightProbe {
  Float32Array SHCoefficients;
  attribute XRRigidTransform offset;
  XRSpace space;
};

If coefficients contains only a single value, it is to be interpreted as an ambient intensity ambiguous of color or directionality.
If coefficients contains at least 3 values, they are triplets of color component values in r,g,b order. Each triplet forms a coefficient of a spherical harmonic (SH) function. The SH function is interpreted as a Pre-computed Radiance Transform (PRT).
The first coefficient (one r,g,b triplet), forms order 0 of the SH function. These first three values can be interpreted as a constant ambient term ambiguous of directionality.
Additional triplets define the coefficients for higher orders, introducing greater frequency and directionality that may be optionally used by more advanced rendering engines.
coefficients must contain the precise number of values required to contain complete orders. Where "n" is the order, (n+1)^2 coefficients are required.

eg:
Order 0: (0+1)^2 = 1 coefficient / 3 values
Order 1: (1+1)^2 = 4 coefficients / 12 values
Order 2: (2+1)^2 = 9 coefficients / 27 values
Order 3: (3+1)^2 = 16 coefficients / 48 values

The color component values of 0.0 to 1.0 are mapped to SRGB color space. Values above 1.0 are allowed in order to express high dynamic range lighting and irradiance values beyond the SRGB gamut.
Rendering engines can choose to either select the nearest light probe for an entire model or to interpolate the lighting components resulting from multiple light probes.
​

Describe the components of Float32Arrays that represent light with red, green, and blue components

As per feedback by @elalish on the initial PR, it is not clear what the individual array elements of Float32Arrays such as indirectIrradiance contain and how they may be scaled.

We need some text to connect the "Physically based units" section to the attributes in the IDL.

createWebGLEnvironmentCube needs to know which GL context to create the texture on

When createWebGLEnvironmentCube is called on XRReflectionProbe, it should include a GL context.

Describe simple renderers that could use HDR cubemap textures and SH lighting

As per the feedback by @elalish in the initial PR:

When an HDR Cube Map texture is available, shadows only have to consider occlusion of other rendered objects in the scene.
@elalish: Not quite sure what this sentence is saying... I'm pretty sure this is hard.

We should add some details explaining how a simpler, non-PBR, renderer (eg, a standalone GLSL shader) could give acceptable results:

Doing this 100% correctly is very hard; however, there are many simple approximations that are commonly used. In particular, it may be sufficient in some cases to combine a baked ambient occlusion map with a IBR shader that needs no realtime dynamic lights.

A simple, non-physically based implementation may simply index the HDR cube map using a surface normal, and blend it with an albedo term representing the color of the surface using operators representing the artists intent.

Update readme to point to explainer and add "unstable" notice.

Now that the PR has landed, there are expected to be some iterations until it is refined enough for an implementation. I would like to update the readme to point to the landed explainer and to add an "unstable" notice to the explainer to ensure that it is not mistaken for a completed spec.

Update affiliation of 'Kearwood Gilbert'?

still in Mozilla?
I suppose we also need to issue IE status... > @AdaRoseCannon

May want to incorporate security/privacy concepts from Ambient Light API

https://www.w3.org/TR/ambient-light/#security-and-privacy

I'm not sure which considerations would apply here, but there may be something worth reusing.

Reference to microphone permissions is confusing

As per the feedback by @bricetebbs on the initial explainer PR, the reference to "and microphone" in the XRReflectionProbe section is confusing.

I was imagining similar UX as requesting microphone permission, not necessarily relating the kind of sensor. It seems that this analogy is adding more confusion than helping, so perhaps we should remove the "microphone" reference.