[Question] Virtual Range finder (Realsense) depth image from Dayflower

[macroing]

[Question] Virtual Range finder (Realsense) depth image from Dayflower

[macroing]

Thank you for showing interest in my project!

You are more than welcome to use this project, with or without modifications.

I am not familiar with virtual range finders, robot simulation systems and Intel RealSense. So I probably have to read more about these.

One way to capture distance as a color, could be to implement your own FunctionTexture. Something like the following:

Texture texture = new FunctionTexture(intersection -> {
    Point3F eye = intersection.getRay().getOrigin();
    Point3F lookAt = intersection.getSurfaceIntersectionPoint();
    
    float distance = Point3F.distance(eye, lookAt);
    
    return Color3F.saturate(new Color3F(distance), 0.0F, 1.0F);
});

1. JavaFX is only used by the application itself, which has some limitations. So the JavaFX Scene class is unrelated to org.dayflower.scene.Scene. The application currently only supports Scenes loaded by org.dayflower.scene.loader.JavaSceneLoader, which is using the Java Compiler API. This means that you can write practically any Java source code when loading a Scene.

2. This project currently has limited support for transformations. At least in the application itself, where it has not been implemented yet. But if you're writing a simple program using the library APIs, you could update the org.dayflower.scene.Transform instance associated with each org.dayflower.scene.Primitive manually. This could be synchronized between each number of render passes.

3. I don't think I have support for that, sorry.

[madhephaestus]

Whoh! If that's all it takes to encode distance as a color, then you DO support "depth camera view"! That feature alone would make all of the code generation needed to convert a javafx Scene into a file loaded by JavaSceneLoader. A friendly suggestion would be to show that off as a highlighted feature.

A very valuable feature would be to let users build Javafx scenes and pass that to a compiler that generates the rendered scene. For robotics being able to pass a 3d environment to code that can generate distance-as-pixels means I could simulate LIDAR sensors, that's a big deal!

[madhephaestus]

Ok, I tried the custom texture code:

Texture texture = new FunctionTexture(intersection -> {
    Point3F eye = intersection.getRay().getOrigin();
    Point3F lookAt = intersection.getSurfaceIntersectionPoint();
    
    float distance = Point3F.distance(eye, lookAt);
    
    return Color3F.saturate(new Color3F(distance), 0.0F, 1.0F);
});

Material materialPlane = new MatteMaterial(texture);

What I expected was a gradient proceeding away from the camera getting darker the further away from the camera it got. Instead its a monotone grey, implying its the representing the distance from the light source instead of from the camera.

[macroing]

I just made some additions to the project.

Two new methods in the Camera class to turn on or off center sampling when generating primary rays. Normally it would be uniformly sampled in the shape of a disk.

I also added a DepthCamera.java scene file. It does work somewhat using both Path Tracing and Ray Tracing. You might have to change the ray depth in the source code, though. I have not yet added support for that in the application (GUI).

Both of these additions will only work in the CPU-renderer. For example, the FunctionTexture cannot be added to the GPU-renderer. That would require a lot of work. I would need a decompiler and some transformations on the decompiled source code, as well as a source code generator to generate the entire GPU-renderer.

I could add a new Texture implementation and add support for it in the GPU-renderer.

Another way would be to create a new renderer type. This might be a better approach, if light sources should be excluded?

[macroing]

I just added a Depth Camera rendering algorithm that you can try, if you want to. It works on both renderers, CPU and GPU. Perhaps it needs some fixes, or parameters that you want to change?

[madhephaestus]

Awesome! The GPU render works, but only after closing the example and opening it back up, maybe an initialization issue or race condition?

I switched to ray casting and it looks close but a little off. The ground closer to the camera than the edge of the sphere should be lighter than the sphere. Think of a ray coming from the camera, and measuring the distance to the object surface its intersecting, the pixel at that point should be the distance as a greytone. Further distance, darker. Parameterize the maximum range would be a bike-shed improvement, but would be really helpful as a configuration.

I also added some more spheres in the line in front of the camera, they should all be different shades, as they are different distances from the camera. In the current version of the render they are all the same shade.

Thank you so much for your expert assistance with this! You rock!

[macroing]

The ray caster works a bit differently. It uses a grayscale color based on the absolute dot product of the ray direction and the surface normal. It's mainly used to speed the rendering up, when all you want is to position the camera somewhere.

When using the first approach above, with FunctionTexture, the path tracer and ray tracer algorithms should be used. But, because they are shooting more rays in random directions and needs light sources to work, the image will have noise in it and possibly shadows.

The new depth camera rendering algorithm, that I wrote about in my last comment, does not need light sources to work. It does not shoot rays in random directions, unless depth of field is used. I might add a maximum distance or scale parameter to it. I currently use a constant local variable called scale, that is set to 0.25.

I have seen a bug where the window is shrinking when you close the tab. This was not the case in Java 8. I just switched to Java 17 and have seen it there. I'm not sure if that's what you're referring to?

[madhephaestus]

OH YEAH! the depth camera is exactly what im looking for!

[madhephaestus]

I can not get over how awesome this is! Yeah, i just checked the depth camera with multiple spheres, and its exactly right, further is darker, with the rings centered around the closest point, this is great!!