0beqz/realism-effects

SSGI, Motion Blur, TRAA - Effects to enhance your three.js scene's realism

Hey! I'm 0beqz and I love working on graphics projects for three.js including:

• SSGI: Real-time global illumination on the web

• SSR: Screen-space reflections to spice up your scene's realism and specular lighting

• TRAA: Cinematic anti-aliasing to make your scene appear more filmic and less pixelated

• Motion Blur: Adds smearing to your screen when moving the camera which gives a cinematic effect often seen in real cameras and games

• SVGF: Modern variance-guided denoiser using temporal reprojection to minimize the noise in raytraced images

• Temporal Reprojection: General-purpose pass that's used to properly reproject the last frame into the current frame as well as deal with all sorts of disocclusion issues to reduce temporal lag - useful for enabling lower per-frame sample counts in effects involving GI, AO and reflections

🎇 What I'm currently working on:

• realism-effects: SSGI, Motion Blur, TRAA - effects to enhance your three.js scene's realism
• screen-space-reflections: Implements Screen Space Reflections in three.js
• enhance-shader-lighting: Better lighting in three.js

🎆 What I've worked on:

• three-billboard-reflection: Performant plane reflections in three.js
• lightmap_interior_test: Three.js lightmap interior test
• env-map-importance-sampling-tiles: Generates tiles of an env map for importance sampling the environment in a Monte Carlo Ray Tracer

👉 If my work helps you and you'd like to support it, then I'd appreciate you donating or sponsoring me. Thanks!

realism-effects's Issues

more loose three peerDep

Since recently, three version was restricted a lot more, is that absolutely required?

realism-effects/package.json

Line 53 in e6e4f57

"three": ">=0.150.1"

I ask this because currently @pmndrs/postprocessing for example only works with r148, but make other libs will benefit from a more loose peerDep?

poissionDenoise.frag compile error if not NORMAL_IN_RGB

If "NORMAL_IN_RGB" is not defined, the following line in "poissionDenoise.frag" causes a compilation error:

float center = texel.rgb;

For a solution see #21

React-three-fiber integration

Is there a recommended way to integrate the SSGI with @react-three/fiber? I tried adapting your code like this, but it did not work:

const SSGI = () => {
    const { camera, composer, scene } = useContext( EffectComposerContext )
    const velocityDepthNormalPass = new VelocityDepthNormalPass( scene, camera )
    composer.addPass(velocityDepthNormalPass)
    const ssgiEffect = new SSGIEffect( scene, camera, velocityDepthNormalPass, options )
    const traaEffect = new TRAAEffect( scene, camera, velocityDepthNormalPass )
    const motionBlurEffect = new MotionBlurEffect( velocityDepthNormalPass )
    return <EffectComposer>
        <primitive object={motionBlurEffect}/>
        <primitive object={traaEffect}/>
        <primitive object={ssgiEffect}/>
    </EffectComposer>
}

Orthographic camera

In the example, when changing the camera from perspective to orthographic, tracing does not seem to work, and because of this, the light creates a completely different picture, slightly overexposed and without such shadows as with a perspective camera.

// const camera = new THREE.PerspectiveCamera(40, window.innerWidth / window.innerHeight, 0.01, 250)
const frustumSize = 10
const aspect = window.innerWidth / window.innerHeight
const camera = new THREE.OrthographicCamera(
	-frustumSize * aspect,
	frustumSize * aspect,
	frustumSize,
	-frustumSize,
	1,
	1000,
)
scene.add(camera)

Perspective (default example)

Orthographic

Is it possible to achieve a similar result with an orthographic camera as a perspective one? Or is it not supported yet?

SSR status

Hello, my statement is more a question than an issue but i didn't know how to reach you so please, forgive me if you find this entry too much intrusive.

Basically, i can't understand the current status of your SSR effect. I mean, i've seen the other repository which at the moment seems to be deprecated due to an incompatibility with the last version of three.js.

At the moment, is the SSR effect implemented in you SSGI code? I've seen the SSR quote in the realism-effects description page but there is no description and so for me is a little bit misleading...

Anyway, thanks for your work, i really appreciate it!

EquirectHdrInfoUniform use worker but not disposed in dispose

I tried to test this pr(pmndrs/react-postprocessing#211) for test in react-postprocessing

I found when SSGIEffect instance created, Javascript VM instance has added so I think it is kind of workers but I can't find where the worker created in chrome Mac OS

Env maps to goes black for other WebGLRenderer with MeshStandardMaterial.

I have 2 WebGLRenderer(Canvas) in my web.
However, The second one didn't make use of realism effects.
But the env map goes black when realism effects were used in the first canvas.

I've dug a bit and found the patches of ShaderChunk.envmap_physical_pars_fragment
(https://github.com/0beqz/realism-effects/blob/main/src/ssgi/utils/Utils.js)

I also found the SSGIEffect is trying to disable the patch at the end of each frame.
(https://github.com/0beqz/realism-effects/blob/main/src/ssgi/SSGIEffect.js)
But that's not working when another renderer tries to use envmap_physical_pars_fragment along the way.

globalIblRadianceDisabledUniform.value = true

this.rAF = requestAnimationFrame(() => {
	this.rAF2 = requestAnimationFrame(() => {
		globalIblRadianceDisabledUniform.value = false
	})
})

Transparent materials not displayed with transparency

When clicking through the sample models on https://realism-effects.vercel.app/, I noted the Swordsman looks wrong around the cape / cutout parts:

I think either transparency or cutout may not be working correctly with SSGI on.

EDIT: even more apparent on "cyber samurai" and can also be seen on "amg" on the headlights, on "motorbike" on the glass parts, on "cyberpunk_bike" on the floor.

I understand there are limitations with depth-based effects and transparency - would be great if those objects would render kind of correct though.

VelocityDepthNormalMaterial incompatible with TransformControls from Three

Thanks for the great project. I've just started using it and spent a lot of time trying to integrate it with react-three-fiber (Success!). Now, I'm just having fun.

TL;DR

I have found the issue that covers this one while I was writing it. :/ Feel free to close. I still think it's good for future reference if somebody has the same issue.

The related issue is https://github.com/0beqz/realism-effects/blob/eb03a9e4a50bd7f0242bafb77d8e9b0c4c73543b/src/ssgi/utils/Utils.js#L238C24-L238C24

Wasted effort

First of all, I'm using realism-effects with react-three-fiber and in general it works just fine. I discovered an issue though, that is architectural.

VelocityDepthNormalPass does deep replacement of materials in the whole scene. That's quite an intrusive operation as it's impossible to tell if some user of replaced material relies on custom property that is went missing.

In my case it's "color" on one of the handlers of TransformControls. The code tries to clone a color without doing undefined checks.

I've temporary fixed it by adding "color" to

const materialProps = ["vertexTangent", "vertexColors", "vertexAlphas", "vertexUvs", "uvsVertexOnly", "supportsVertexTextures", "instancing", "instancingColor", "side", "flatShading", "skinning", "doubleSided", "flipSided"];

Proxy can be used to fix at least part of property access errors but it is still going to be brittle. In my opinion it's good to implement both approches.

Partly related #14 as it will allow to exclude some of the offending objects in scene graph.

Relevant code

TransformControls:
https://github.com/pmndrs/three-stdlib/blob/4c04593ee49bb0b022025718844f3ce2b21f67bf/src/controls/TransformControls.ts#L1301C40-L1301C40

ssgi/.../Util:
https://github.com/0beqz/realism-effects/blob/eb03a9e4a50bd7f0242bafb77d8e9b0c4c73543b/src/ssgi/utils/Utils.js#L238C24-L238C24

Trailing effect when rotating camera

continuing this thread from twitter https://twitter.com/vis_prime/status/1632602124269023233?s=20

I'm noticing trails being formed which does not happen on the demo page

2023-03-08.16-02-07.mp4

HBAO: black spots issue (possibly NaNs or similar)

See: https://twitter.com/edankwan/status/1659655493302796288?t=pWtM2sTx9CpX_W_mYfmvQg&s=19

Vertical objects appear with strange noise

realism-effects： Memory leak problem

I tested both locally and online, with your example code, as well as my own working code, and experienced a memory leak: the same model, loaded multiple times, and each load cleared the memory of the previous model, but the memory continued to rise. When I use normal rendering, this problem does not occur. The memory comparison diagram is as follows: The leak points to Realistic-Effects, and I think it may be that the Realistic-Effects rendered real particle geometry memory is not fully cleared

Missing VelocityDepthNormalPass?

Hey, congrats on this release! I'm excited to give it a quick try at least, but it seems the VelocityDepthNormalPass is currently not being bundled/exported?

Heat spot on perfect reflection.

I've noticed high heat spot artifacts when perfect reflection reflects each other.

Changing color in the material on runtime won't work

Changing "color" in the material on runtime won't work.

It seems like propsPrimitive didn't have color property inherited from Material class.
(https://github.com/0beqz/realism-effects/blob/main/src/gbuffer/material/GBufferMaterial.js)

let props = Object.keys(gBufferMaterial)

props = props.filter(
	key => !key.startsWith("_") && !key.startsWith("is") && key !== "uuid" && key !== "type" && key !== "transparent"
)

const propsPrimitive = props.filter(
	key => typeof gBufferMaterial[key] === "string" || typeof gBufferMaterial[key] === "number"
)

Support for mobile devices

First of all, amazing what you have been doing here! Thank you for all your work here.

I was wondering if there will also be support for this on mobile. I tested with the AO effects and there seem to be issues on my Android device. Do you know a workaround for this?

OffscreenCanvas in demo issue

Using OffscreenCanvas seems to result in issues when using Safari, so it should maybe be removed from the demo.

Glitches

Hello! I have a problem on my ipad pro with m1 chip when i bring the camera close to an object

Video

SSGI is not working with some material and texture properities

Just testing a bit on adjusting texture and material with realism effects SSGI passes
I notice the texture repeat, center, rotation and offset is not working, also in material normal strength, env map internsity also not working

Reduce trailing effect for a larger scene

Although, am not using any environment map for the scene, I quite love the way SSGI lifts up the scene. Is there a way to avoid the trailing effect at longer distance?

Am using this settings as option

options = {
  "distance": 1.5,
  "thickness": 20,
  "autoThickness": false,
  "maxRoughness": 1,
  "blend": 0.95,
  "denoiseIterations": 1,
  "denoiseKernel": 3,
  "denoiseDiffuse": 20,
  "denoiseSpecular": 0,
  "depthPhi": 14.022,
  "normalPhi": 0,
  "roughnessPhi": 0,
  "envBlur": 0.55,
  "importanceSampling": true,
  "directLightMultiplier": 1,
  "maxEnvLuminance": 50,
  "steps": 8,
  "refineSteps": 1,
  "spp": 3,
  "resolutionScale": 2,
  "missedRays": false
};

simulation_20Mar.mp4

Build via Github Actions and publish up-to-date preview/production deployment builds to npm

Hi,
First-of-all, thanks for this great library of effects! I'd like to try-out some recent realism-effects which are not yet available in the module version published to npm 1.1.2. I see you have deployments set-up for production/preview - which might just be hooks to vercel, and publish up-to-date demo pages after any commit.

Could you also push the updated realism-effects build to npm within the same vercel deployment task? Or set-up a Github Action to do so, eventually on versioned? Would be great to test new featues without building the lib ourselves.

Using environment map won't work in the MeshStandardMaterial after applied realism effect.

Here is a minimal example.
https://codepen.io/benzsuankularb/pen/WNLzMyg

Basically, I've setup 2 canvas
One with Realism-Effect and one with normal MeshStandardMaterial.
If both canvases are enabled then the env map in MeshStandardMaterial won't work.
Both canvases didn't share any resources on the application layer.

I'm not sure if this is some issue on the Three / WebGL layer.
Or there are some modifications on the Realism-Effect applied to the MeshStandardMaterial shader.

SSGI with Three Js Lights

Hi @0beqz ,
Thanks for making the realism-effects. SSGI works great with HDRI images.

Does it support three js Lights? I tested it with three js Sunlamp, SSGI does not seems to work.

Support multiple materials per mesh

While the GLTFLoader always guaranteed only single material per mesh.

Multiple materials per mesh are natively supported by three.js.

It'd be nice if we cloud add this support. :)

Use "Selection" from `postprocessing` to control which objects participate in SSGI

So helpers like TransformControls crash the render loop (same issue as the old SSR )

and THREE.Lines create black overlay/glitch problems when SSGI is turned on

So if we can use the selection class from postprocessing that would be nice to have eliminate these problems

Emissive objects?

Hey there! Thanks for releasing such an amazing project!!

Are emissive textures supposed to work? They seem to have no effect, even when creating some random messages to test it out.

I'm trying to get a video texture to reflect on the floor like in one of your old examples but nothing seems to pick up light from emissive textures.

I just want to make sure I'm not missing something!

NaN Propogation Error

If you run the demo and change the model to cyber_samurai, the whole model turns black within a few seconds.

I am running this on Mac OS 13.5.1 with M2 chip and google chrome 116.0.5845.140. Here is the

nan_prop.mov

Consistent AO depending on scene scale

Hello, I'm having an issue setting up the AO (both HBAO and SSAO) with varying scene sizes.

In this first example, I have a pretty standard setup with the HBAO effect. Everything works as expected.

When I now introduce a scalingFactor that scales all elements in the scene as in this second example, the results of the AO are not consistent. I tried to adjust the distance, thickness and radius by the scalingFactor with no success.

Is there a way to achieve consistent AO results depending on the scene scale?

Anti-Aliasing Bug?

There seem to be some weird layers on the model? Interestingly, the problem gets resolved only when SMAA is applied.
I'm curious if you're aware of the underlying cause for this occurrence.
Your insights would be greatly appreciated :)

Set clearcolor?

First of all: great work! Thanks.

Perhaps I'm missing something, but would it be possible to make it adhere to renderer.setClearColor() or canvas.style.background?

Seems that without the dome mesh the background is always black:

Setting scene.background to white yields artefacts in the corners:

Request for Tutorial or further Documentation/Example

Difficult to setup and integrate into an existing three.js project. Setting up the Offscreen Canvas to use elsewhere is not explained. Also doesn't seem to work at all on mobile because of this?

"doesnt support camera of type "perspectiveCamera"

Hi, great work on this!

I have the following code

const camera = new THREE.PerspectiveCamera(40, window.innerWidth / window.innerHeight, 0.01, 250)
const velocityDepthNormalPass = new PostProcessing.VelocityDepthNormalEffect(this.getThreeScene(), camera);

I am currently getting the following error

I have no doubt that it is completely my problem....however, I am stumped at where to start to troubleshoot it so any suggestions would be very much appreciated. Im using Threejs r155