Comments (6)
Hiya!
We don't currently support visual observations, but this is on our todo-list, as it's a fairly common ask. Will update this issue once in lands, probably within the next month.
from brax.
Hello,
Just posting to mention that I'm also interested in having visual observations -- ideally rgbd -- thanks for your work!
Martin
from brax.
sounds good
from brax.
if you want to test the trained model in the real world, its very hard to setup the realworld-camera in the same angle/position as the camera in the simulation.
It would be nice if the user could e.g. define a sphere/box around the simulated-camera-placement and in training the simulated-camera-position changes slightly inside the defined sphere/box. So the model will get more generic.
Perhaps the user need also to define a e.g. arrow that points from the sphere/box to the point-of-interrest and then the camera-position can change but it still knows where to point to the camera.
Another thing with camera too would be to add a plane where pictures can be displayed. These planes could then be used as ground or walls, so that in every new training episode another picture could be displayed and the camera can do Domain-randomization, also nice for generic models.
from brax.
Same here. This feature would be great! Also great job folks!
from brax.
There is a slow cpu camera implementation that you could try out, based on PyTinyRenderer. This is the same renderer using in PyBullet, when no GPU OpenGL/EGL hardware is available.
PyTinyRenderer can render RBG, depth and segmentation mask. It supports cubes, capsules and general triangle meshes. Texture mapping and shadow map. The renderer is not anywhere close to Unity or Unreal but maybe sufficient for some learning projects.
See https://github.com/google/brax/blob/main/brax/io/image.py and this colab, scroll down to the render image function.
a colab how to use pytinyrenderer standalone.
Closing for now. If this doesn't work for you, please re-open.
from brax.
Related Issues (20)
- Pusher environment with Spring pipeline HOT 3
- Optimizer with MultiTransform throws ValueError HOT 2
- Documentation bug: wrong observation and action spaces ordering HOT 3
- `mjx.ncon` removed as of MuJoCo 3.1.5 HOT 2
- policy callback 'policy_params_fn' for other algorithms? HOT 1
- ptxas version missmatch HOT 1
- CUDA OOM with jax/pytorch notebook HOT 1
- very very slow on local computer (even with GPU) HOT 2
- What replaced the old "pmap.is_synchornized" called in new brax versions? HOT 1
- Jacobian of State Dynamics HOT 3
- how to do n controlled physic steps per every control step
- NaNs at Inference HOT 7
- Setting Initial Camera Position in Brax Visualizer
- Rendered Plane Texture Issue in Visualizer
- Brax's Simulator Engine Swap
- TypeError: RandomNumberGenerator._generator_ctor() takes from 0 to 1 positional arguments but 2 were given
- Domain randomization with mjx backend
- Reference backend for RL literature
- Wrong flag definition in `brax.training`
- Error When Rendering Scene Without Contacts: AttributeError in _compress_contact()
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from brax.