placing camera in simulation world

[flobotics]

placing camera in simulation world

[cdfreeman-google]

Hiya!

We don't currently support visual observations, but this is on our todo-list, as it's a fairly common ask. Will update this issue once in lands, probably within the next month.

[mweiss17]

Hello,
Just posting to mention that I'm also interested in having visual observations -- ideally rgbd -- thanks for your work!
Martin

[flobotics]

sounds good

[flobotics]

if you want to test the trained model in the real world, its very hard to setup the realworld-camera in the same angle/position as the camera in the simulation.

It would be nice if the user could e.g. define a sphere/box around the simulated-camera-placement and in training the simulated-camera-position changes slightly inside the defined sphere/box. So the model will get more generic.

Perhaps the user need also to define a e.g. arrow that points from the sphere/box to the point-of-interrest and then the camera-position can change but it still knows where to point to the camera.

Another thing with camera too would be to add a plane where pictures can be displayed. These planes could then be used as ground or walls, so that in every new training episode another picture could be displayed and the camera can do Domain-randomization, also nice for generic models.

[fgolemo]

Same here. This feature would be great! Also great job folks!

[erwincoumans]

There is a slow cpu camera implementation that you could try out, based on PyTinyRenderer. This is the same renderer using in PyBullet, when no GPU OpenGL/EGL hardware is available.

PyTinyRenderer can render RBG, depth and segmentation mask. It supports cubes, capsules and general triangle meshes. Texture mapping and shadow map. The renderer is not anywhere close to Unity or Unreal but maybe sufficient for some learning projects.

See https://github.com/google/brax/blob/main/brax/io/image.py and this colab, scroll down to the render image function.

a colab how to use pytinyrenderer standalone.

Closing for now. If this doesn't work for you, please re-open.