Sporadic gaps / precision errors in tiles.

[Revmatek]

Sporadic gaps / precision errors in tiles.

[daleeidd]

That's okay. Thank you for your help so far.

[daleeidd]

Thanks for letting me know. That should help narrow down a solution. And good to hear you have a workaround.

[Revmatek]

The following was produced by starting and stopping play mode over and over in the BoatWakesScene. This may be a bad state transitioning between playmode and editor or it may be directly related to what I am seeing in playmode sporadically.

Here is a screenshot that was very difficult to get, involved recording and going frame by frame:

This is not the exact way I saw it before but I am trying to find a reliable way to reproduce it. It does look very similar if not identical, hard to say given I have not captured a frame of it happen sporadically. I can can't say for certain its the same issue.

[Revmatek]

I have seen this happens a couple more times while doing unrelated things. It doesn't seem like tiles being misplaced or culled or even the position. It looks like for a fraction of a second that a tiny chunk of tile is gone, like a Tetris piece, I don't know how else to describe it, but its very noticeable, its like everyone now and then some sort of random rectangular polygon disappears. Its not clear to me whether this is caused by occlusions or culling of some kind, all I know currently is that it did not happen in previous versions. I will continue to make an effort to record it, but I'm not sure how that would help if you can't reproduce it, which the only way seems to be to stare at the screen while doing other things for extended periods (i.e. playing a game)

Gonna try updating my drivers, maybe it will go away. 😵‍💫

[Revmatek]

The following was produced by starting and stopping play mode over and over in the BoatWakesScene. This may be a bad state transitioning between playmode and editor or it may be directly related to what I am seeing in playmode sporadically.

Here is a screenshot that was very difficult to get, involved recording and going frame by frame:

This is not the exact way I saw it before but I am trying to find a reliable way to reproduce it. It does look very similar if not identical, hard to say given I have not captured a frame of it happen sporadically. I can can't say for certain its the same issue.

Pretty sure this is something different and not the problem.

[Revmatek]

Gonna try updating my drivers, maybe it will go away. 😵‍💫

Still occurs after driver updates.

[daleeidd]

Thanks for reporting. It is definitely a precision issue so drivers will unlikely fix it. I don't have the time to look at it right now. Having the camera transform and Ocean Renderer settings when it occurs would certainly help although I understand it may be hard to obtain the former.

[Revmatek]

Thanks for reporting. It is definitely a precision issue so drivers will unlikely fix it. I don't have the time to look at it right now. Having the camera transform and Ocean Renderer settings when it occurs would certainly help although I understand it may be hard to obtain the former.

I have seen this happen at least once in the BoatWakes scene so the settings there should work to reproduce. Should not require any changes to reproduce, the problem is it seems to be some sort of random timing issue that is difficult to capture.

[Revmatek]

I have some new information after trying to reproduce this again, I noticed the tiles that are being culled (still haven't seen it happen)

I found the squares that are disappearing but I don't know how to describe their function (depth, shadow, maybe something to do with the water interaction around the sides of the boat)

In this image below you can see strange artifact like tiles that don't seem to match up with each other

above view for context:

Pressing the windows key temporarily disables the shadow area / tiny tiles. I am confident that these are the tiles I see being culled momentarily.

More images to better show tiny tiles:

Here I have increased the lighting / exposure to better show the strange tiles:

Whatever is happening is happening near this small tile here:

In wireframe I can see that the smallest tile is being broken up into polygons that resemble what is being culled/disappears:

I can better help troubleshoot this with the following information:

What are these tiny tiles that don't match up, are they artifacts? (its obvious this is part of the LOD, Mesh, Camera distance etc.)
How can I change their behavior?
Is there a setting related to this?

The last few times I saw part of the ocean disappear it was in this small area in seemingly random configurations of cubes/tiny tiles. I am not sure if that means it cannot happen to longer sections of the ocean or not (maybe if they also have similar tiling).

Selecting the ocean renderer in the inspector at runtime makes them much more visible (because the game/fps slows down maybe)

Seems unrelated to subsurface scattering as I can still see the tiles with it turned off:

The artifact themselves do seem affected by "Shadowing" on the material/shader.

[Revmatek]

@daleeidd I am very excited because while trying to narrow this down I was able to capture this illusive bug on video.

Here is a screencap of the original problem. Its looks like the tiles are physically separated.

Here is a clip of the bug. This occurred roughly 5 minutes into me playing with various inspector settings trying to find the problem. You can also see the tiny tile artifacts just prior, though that may be unrelated. The FPS in this video dips several times due to messing with the inspector.

TileBug.mp4

[Revmatek]

I am looking into causes based on this : #1135 (comment)

Definitely more than a subpixel.

The only maybe related changes I see are here:
https://github.com/wave-harmonic/crest/commits/master/crest/Assets/Crest/Crest/Shaders/OceanVertHelpers.hlsl

I see various commits that look like they might have changed the behavior for what would have been included with my update from 4.15.2 to 4.17.

Without a complete understanding, this was seems like its around the time frame.
95e636e

I am a bit confused looking at the history for the precision fixes, some of the changes seem to disappear from 2020 to 2023, maybe here:
c6f8160
Which I think is part of the second precision fixed you mentioned above.

[Revmatek]

#1136 (comment)

This now seems to me to be directly related given the similar displacement of tiles. Its possible that any fix for the record problem would also fix this.

Definitely some weird timing issues going on. I can't get this start/stop play issue to happen in latest with a standalone project
using

But if I import the same files into my project (I guess which makes play mode start slightly slower) than I can see the gaps. Feels like I am chasing ghosts.

[daleeidd]

Shadows can look strange due to vertex interpolation but that video looks a bit more strange than usual. On the material there is a "Debug Visualise With Colours" dropdown. It has a shadow option which may be easier to see what is happening.

As for the gaps, I wonder if it is triggered by a scale change. If you could set both the Min Scale and Max Scale on the Ocean Renderer to the same value to prevent scale changes and see if it comes up again. The water won't look correct with this setting but if you hit the problem frequently it should be quick to confirm.

[Revmatek]

Min and Max where set to 4 and 256. I tried setting them both to 1, 4 and 256 and various other different values. I did not see any gaps occur. I assume the goal was to see gaps appear frequently which was not the case so I did not wait several hours.

I did play with some other settings, the only one that had an effect was LOD Data Resolution, I was able to produce similar looking gaps to the video that are there constantly, i'm unsure if this is a hint or just a bad value. The current value is 384.

I set the value to 496 (seems to happen with anything over 400) and these gaps appeared:

To me, the pattern of the gap moving to the right seems similar to the recording. I was above the water (in a plane) looking down the other day and I saw something similar.

Changing the LOD Count did not seem to affect that gaps. I did also see different looking gaps when changing just Geometry Down Sample Factor to 16, but I am not sure its related:

[daleeidd]

The resolution needs to be at certain intervals (I believe 128 is a safe number) otherwise issues will appear.

The goal was to see if they wouldn't appear like they have been which would make me think that scale is a factor. If scale is at play then will make it much harder to reproduce.

[Revmatek]

I will keep an eye out for it occurring again, but I am not sure what else I can do. There is definitely a bug(s) somewhere. I did a clean install and reintegrated and have not seen the issue occur all week doing other things. It seems like there are a handful issues at play.

[Revmatek]

I saw this happen again while above the ocean and entire section (large slice) of the ocean disappeared in the same manner. I do not believe this is due to any inspector settings given the clean state with latest and it occurring in examples scenes, seems like it is most likely a problem with timing / execution order / code somewhere. Something that was not in earlier versions or something that was added that changed the timing of other functions. I would bet these momentary disappearances are related to a number of similar tile gaps that appear while doing various things in the editor (changing scenes, entering/exiting playmode, Window key behavior). There are times in the editor where it no longer draws the ocean, if those problems are fixed I think this problem may become more apparent or disappear entirely as it may be a symptom of a larger drawing problem.

[Revmatek]

@daleeidd Could this be caused by inaccuracy with high far clipping on the camera such as:

?

This is something I changed / allowed around the same time I updated the crest version. Can be anywhere from 2000-500000 with in game camera settings and code. I added some dynamic setting of the far clip plane to reduce other issues underwater and limit it unless at very high altitudes roughly 2000 - 10000 meters, somewhere in this range the far clipping needs to go up to prevent things from not looking great. There is an obvious need to see very far in some cases when in an infinite ocean though. It occurred to me that this may be the same cause since it is similar to other culling issues I was having underwater.

I was able to changed my LOD back to 7 without issue, I still saw the problem once or twice (maybe less frequent, idk), I will have to wait a bit and see if it still shows up keeping view distance below 100000 unless at a view high altitude.

[daleeidd]

I see. Curious to see if that is related. I tried 500k but didn't experience any problems.

[Revmatek]

I see. Curious to see if that is related. I tried 500k but didn't experience any problems.

I still see it occur occasionally at 100k. Not sure if less frequent or even related. Not something that is immediately apparent. Takes hours or days to see it.

[Revmatek]

@daleeidd After doing another deep dive, I am fairly confident this is being caused by

256 and 512 do not seem to cause problems and I have not seen the gaps in weeks. I cannot be sure it's gone though so I will keep monitoring. However, I can say for certain debugging again led me to this code which I was looking at last time I looked into this:

I focused on the grid size this time, its difficult to debug this stuff with shaders so I don't have info on the values in the helper:

having 384 (128 + 256), presumably an acceptable fraction based on the comment seems to produce long repeating fraction floating numbers where 256 and 512 produce nice even whole numbers of some sort. This is what led me to believe something was wrong there.

Hoping this solves the issue. I will let you know if I see it again, but it has been a while.

[Revmatek]

Just an update on the workaround and for your info, it does appear to have a positive affect. I have not seen any close up tile gaps anymore. I do occasionally when flying through the air see very quick gaps in further away tiles but it is no longer critical / very obvious. It also seems to be far less frequent.

I do think there is still a precision issue going on that may not be fixed by settings alone.

[Revmatek]

I still see this happen (I just saw it). I can't be sure it is happening less, I seem to see it up close less after the changes, but it is still a serious problem, I mostly ignore it as much as I can tolerate. I at least find it very noticeable because I see it every time it happens now, someone else might not notice, but I have seen it on other peoples machines. It will just start happening after a certain amount of time or instantly (seems random). Could this have something to do with garbage collection or something? Any ideas? How could it be happening when the camera is not moving (or at least very slightly with buoyancy), just staring at the ocean from some perspective and all the sudden a gap in the tiles appears. It doesn't make a lot of sense to me. I have seen it happen in the demo scenes but I cannot capture it, you just stare at it and it breaks after a while and then corrects itself quickly within a frame, doesn't seem to matter what I am doing or what the settings are, it will just create a gap somewhere in the tiles either close up (which doesn't seem like a precision error) or long / far away along some of the longer tiles. Is there some key piece of information related to how the tiles positioning themselves that I am unaware of? Something that might give me a hint as to where to look? It seems like there is various code related to precision that are magic numbers to me, can you help me better understand why it is setup this way, perhaps I will be able to identify the problem. I am almost certain this did not happen in previous versions (4.17->4.19), I would have surely noticed it. The only other thing I can think is that somehow something I am doing with the position of the ocean is no longer compatible with the ocean precision issues and that whatever I saw previously in the BoatWakes scene was an unrelated problem.

This is the stuff that is confusing to me (along with other locations where magic numbers (:OceanGridPrecisionErrors) are being used to avoid some sort of apparent precision issue with GPUs / vectors? but why would such things happen up close near 0,0,0 and not far away where floating points start to break down):

either not solve gaps at large distances or introduce too many overlaps at small distances that sounds a lot like what I am seeing. at least in terms of where I am see gaps at large and small distances.

Where do these numerical errors come from?

I do change the positions of the ocean for tides and buoyancy queries which require the ocean to move. I am noticing that the root movement is very precise, so maybe I need to completely avoid moving the ocean and try to use the root instead... see if that has an effect on the gaps and the ocean itself will only move with the floating origin. I can only assume that moving the root y is not related and I should be ok for tides, it may be my use of x, z that is the problem.

[Revmatek]

Has nothing to do with any scripts I have moving the ocean renderer (unless it is Y / tide), I eliminated those and it occurred almost immediately when I tried to return the LOD data resolutions back to 384 from 512.

If I leave the OceanRenderer with min scale and max scale at 32 what drawbacks will occur? scale ocean mesh based on camera distance to sea level, to keep uniform detail. It seems like I cannot do this as it changes the appearance of the ocean negatively. Perhaps I can hardcode it for the editor only.

[Revmatek]

@daleeidd Here is a video of reproducing this in the BoatWakes scene in latest. It is similar to what I mentioned I saw in the boat wake scene previously but was unable to capture. It is also near identical to what I see flying / boating around sometimes, but somewhat dissimilar from this video #1136 (comment)

Gap.mp4

Loop (seems to occur for only 1 frame before correcting itself)

gap_loop.mp4

I included some camera info in the video, but I don't think it is related. I had a Scale by factor wave input at the origin in an attempt to simulate my scene with the drydock, but I also don't think that is related. I attempted to do this multiple times and was unable to see it again or capture it with the root scale displaying. I also have the far clipping set very high because I thought maybe that would exaggerate the issue and make it easier to reproduce, it does not seem to. In my scene the far clipping was 100000 when I last saw it.

Here are the ocean settings:

I was trying to reproduce this under the theory that this is in fact caused by scaling somehow so I was performing sort of a cosine wave / zig zag on the surface of the water to force the scale to bounce between 2-16 ish.

I have been developing with:

#if UNITY_EDITOR
        OceanRenderer._lodDataResolution = 384;
        OceanRenderer._maxScale = 32;
        OceanRenderer._minScale = 32;
#endif

and have not seen the issue since. But it may take more time before it pops up again.

[Revmatek]

I was poking around trying to figure out where the problem is coming from and I ended up in the same place as last time #1136 (comment)

I played around with the value going in there to see if I could force it to create similar errors.
Adding some random error seems to produce very similar results, though I would not say they are identical, sometimes it looks pretty close. So maybe there is some error that can occur when the camera and tiles are in a bad sweet spot.
instanceData.Current[lodIdx]._geoGridWidth += Random.Range(-0.0001f, 0.0001f);

Similar looking gaps:

Does the root of the tile need to be at the exact position of the camera minus the :OceanGridPrecisionErrors stuff? Would snapping it to a grid do anything or is that exactly what the precision stuff is trying to avoid?

Example:

pos.x = Mathf.Round(pos.x / 1) * 1f;
pos.z = Mathf.Round(pos.z / 1) * 1f;

(Update: Does not appear to be)Is it possible the :OceanGridPrecisionErrors stuff is causing this? How do I reproduce the issue that this code is meant to fix? I could disable it and see if the problem goes away, but I also don't want to re-introduce whatever this was meant to fix. It does not appear to cause any harm in the demo scene when I comment it out from the 3 locations, but Idk what I am looking for.

[daleeidd]

This is the stuff that is confusing to me (along with other locations where magic numbers (:OceanGridPrecisionErrors) are being used to avoid some sort of apparent precision issue with GPUs / vectors? but why would such things happen up close near 0,0,0 and not far away where floating points start to break down):

Think of it as a rounding error. Round(0.5)… does it go up or down. In this case it can be either so we avoid those positions. This is likely the problem, that is that the solution is not complete as the comment points out.

either not solve gaps at large distances or introduce too many overlaps at small distances that sounds a lot like what I am seeing. at least in terms of where I am see gaps at large and small distances.

I do not think it is related based on your videos. This is more for fine gaps between tiles. Not the ones you are seeing.

Where do these numerical errors come from?

Floating point numbers have precision issues.

I do change the positions of the ocean for tides and buoyancy queries which require the ocean to move. I am noticing that the root movement is very precise, so maybe I need to completely avoid moving the ocean and try to use the root instead... see if that has an effect on the gaps and the ocean itself will only move with the floating origin. I can only assume that moving the root y is not related and I should be ok for tides, it may be my use of x, z that is the problem.

Moving the ocean renderer vertically should be fine. Changing to the root will not change anything. You should not move the ocean renderer on the XZ other than by moving the viewpoint.

If I leave the OceanRenderer with min scale and max scale at 32 what drawbacks will occur? scale ocean mesh based on camera distance to sea level, to keep uniform detail. It seems like I cannot do this as it changes the appearance of the ocean negatively. Perhaps I can hardcode it for the editor only.

Very necessary for visuals when camera is high.

I have been developing with:

#if UNITY_EDITOR
        OceanRenderer._lodDataResolution = 384;
        OceanRenderer._maxScale = 32;
        OceanRenderer._minScale = 32;
#endif

and have not seen the issue since. But it may take more time before it pops up again.

Then it is likely to do with scale changes.

Does the root of the tile need to be at the exact position of the camera minus the :OceanGridPrecisionErrors stuff? Would snapping it to a grid do anything or is that exactly what the precision stuff is trying to avoid?

It does or the centre of detail will be off plus you will see the snapping.

(Update: Does not appear to be)Is it possible the :OceanGridPrecisionErrors stuff is causing this? How do I reproduce the issue that this code is meant to fix? I could disable it and see if the problem goes away, but I also don't want to re-introduce whatever this was meant to fix. It does not appear to cause any harm in the demo scene when I comment it out from the 3 locations, but Idk what I am looking for.

I already linked to the relevant PRs.

[Revmatek]

I did just confirm that this can still happen even when the minScale and maxScale are the same. I have had it set to 8 for several hours and just saw it occur. Any other ideas I can try?

[daleeidd]

Nothing comes to mind.

[Revmatek]

Here is some more video / info on this:
Video:

Gap.1.Compressed.mp4

Frame capture:

I have been looking into maybe it being caused by some sort of execution order problem, but that does not seem to be the case. It is almost like something is incorrect in LateUpdate and then corrects itself on the next frame, but as far as I can tell there is no way to aggravate the issue or cause it to happen. The only thing that has an affect is the _lodDataResolution which only moves the problem to tiles further away I think. It seems completely random and can happen at any moment within seconds or hours. I happened to capture this one while doing something else.

[Revmatek]

I saw this happen 3 times yesterday over the course of 12 hours or so. They were all different tiles at different distances. It happened 1 time normally as described previously, 1 time looking up from underwater, and one time while the camera was idle and mostly not moving (so really had nothing to do with anything that was happening in the scene). They seemed evenly spaced, which is weird. Makes me think it is related to timing or a delta somewhere that is rolling over or resetting to 0 after 4 hours. I was looking for anything that jumped out at me like that but did not find anything obvious that caused similar gaps when intentionally changing. I did however notice something else.

I was once again looking at SnapAndTransitionVertLayout which I had previously introduced random values into and saw similar gaps so I assumed something was going on there or was somehow related. I also noticed this which for whatever reason seemed more relevant:

io_worldPos.xz -= frac(objectPosXZWS / GRID_SIZE_2) * GRID_SIZE_2;

the important part being the comment // caution - sign of frac might change in non-hlsl shaders

Presumable the result of frac is always supposed to be positive? I would think so but idk. I looked around for information on this and found people talking about strange results and negative values in some cases. So what happens when this is negative? It produces gaps near identical to what I saw for a single frame for a single tile, but every frame for every tile.

So this produces gaps everywhere for obvious reasons
io_worldPos.xz -= -frac(objectPosXZWS / GRID_SIZE_2) * GRID_SIZE_2;

So then my question would be what can cause this to be negative? The documentation I found did not seem to indicate sign. The answer I guess is not important if it fixes the problem, which I will not know until hours or days from now.

Assuming that this somehow returns a negative value occasionally:
io_worldPos.xz -= abs(frac(objectPosXZWS / GRID_SIZE_2)) * GRID_SIZE_2;

I tested this with:
io_worldPos.xz -= abs(-frac(objectPosXZWS / GRID_SIZE_2)) * GRID_SIZE_2

and it seems to work normally, but idk yet. It is interesting and makes a bit more sense to me than some of the other things I looked into so I am hopeful.

[Revmatek]

#1136 (comment) does not seem to completely fix the problem. Gaps seem different (maybe smaller) and close but are still there. Seems likely there is something wrong with SnapAndTransitionVertLayout

[daleeidd]

Does it seem like an improvement? Did you abs both of the fracs?

[Revmatek]

Does it seem like an improvement? Did you abs both of the fracs?

No it does not seem to have an effect, at least not one I can measure. I believe so. I went through several iterations of frac/abs combo none seem to work. I had to wait several hours between attempts while doing other things, but I usually see it when it happens.

I was referencing this:
https://developer.download.nvidia.com/cg/frac.html

I don't think I can tell that even if it was returning a negative value that the value it would have returned was correct. However I made some changes based on the reference implementation and tried v - floor(v); for various things and it had the same effect, meaning that everything looked the same and gaps still appeared.

I did notice something today though, it seems like it can be aggravated to occur more frequently. Though I am not sure how, I have just observed that when running multiple instances of Unity (for multiplayer testing) that as the game is loading on one instance the already loaded instance is far more likely to have gaps appear. Seems like a 50/50 if I see it on the other instance while loading. They are the same gaps, just more frequent. So maybe the problem is more apparent when there is more strain on the cpu/gpu while loading scenes. (I normally see this at 60fps though). I tried limiting the fps in the demo scene and creating a script that does a bunch of cpu work but that did not generate any gaps. This behavior may explain why I was able to capture it so easily here: #1136 (comment) you can see right when it occurs it dips from 60 to 6-7 fps. I think at the time this was occurring because I had lots of vegetation loading in the background and I had the OceanRenderer selected in the inspector which caused a heavy performance impact.

It is definitely some sort of timing issue maybe based on the execution order and Unity changing something or something changing for a single frame and altering the outcome of a single snap in some way. Based on the frac results and the logic in there is must be somehow nudging in the wrong direction for a single side of a tile. Very perplexing.

[Revmatek]

Side note which may be related. I do see this sometimes in the editor when compiling code after disabling these: #1124 (comment)

The gaps seems related, but disappear when unity finishes whatever it was doing reloading code. These are also the gaps that appear if the frac is negative.

It is more similar to the frac offset not being applied at all:

Though I cannot be sure it is related, these issues did appear at the same time after I updated crest.