Env.step() with no action

[openai]

Env.step() with no action

[keleftheriou]

I've only been playing with the 'CartPole-v0' environment so far, and that has an action_space of spaces.Discrete(2) which led me to my comment.

I wonder if making Env.step() have action=None as a default could work, letting the environment complain if it can't accept it. The API would be more general this way, and the potential for API misuse by either environment users or environment authors seems low.

Alternatively, 'CartPole-v0' could just be changed to take a Box action_space instead of a Discrete one, although it feels like this particular environment might have been purposefully made to not take variable force values.

Good to know about the mujoco environments utilizing a notion of "effort" through the reward returned, thanks.

[jonasschneider]

This seems like it's mostly a semantic issue with env action spaces. In some envs, "not acting" is meaningless -- take board games like Chess or Go (the latter we have as an env already) for example: if it's your turn, you can't meaningfully choose to "not act".

Also, efficiency/effort are ill-defined concepts whose semantics depend on the env. In fact, what you propose is already implemented in some envs. For example, the mujoco envs penalize you (i.e. reduce reward) based on the joint torques that you specify, i.e. what could intuitively be described as the "effort".

So it seems like this is a per-env semantic issue. We're happy about proposals for definitions of efficiency for specific envs, though!

[tlbtlbtlb]

In (I think) all the physics environments the actions are converted to torques, so a vector of 0s is a reasonable least-effort action. That might not be obvious from the action_space, which usually is a Box with low=[-1, ...] and high=[+1, ...].

[gdb]

The feedback makes sense; thanks for the input!

This isn't part of the standard formulation of RL, so I think for now we shouldn't add it (and I'm not sure if it'd make sense for every env). I'm going to close for now, but if we see more demand for something like this, we'll revisit!

[ivanvoid]

@gdb so you wanna say this issue was never resolved?
Yea 4 years later, please add option to stay still in 'CartPole-v1',
or just help to figure out how to make it possible, this also will be really appreciated.

This option really integral for development of Event-Triggered Control RL systems,
I would say this would help a lot making this Cart Pole a benchmark for those applications.

[soroush-bn]

@gdb I came across the same problem when I want to add additional actions for "Taxi" environment, I want to add diagonal movements to the grid world, however, it is not possible to perform such action when agent is on the edge of the grid, thus, whether I need to create my custom env or use a "NOOP" action and pass it to env.step() function to see the observation and etc.