Cartpole observations can occur outside of observation space limits

[openai]

Cartpole observations can occur outside of observation space limits

[gris-martin]

I get the same problem with the lower (negative) x-coordinate.

[rafaelcp]

It's still happening:

2016-05-26 03:10:56,776] Observation '[-2.41935746 -1.66550681 0.00714052 -0.11863974]' is not contained within observation space 'Box(4,)'. [2016-05-26 03:10:56,835] Observation '[-2.43129996 -1.78584844 -0.01847349 -0.11857105]' is not contained within observation space 'Box(4,)'. [2016-05-26 03:10:56,856] Observation '[-2.41582085 -1.13530746 0.07019884 0.22842913]' is not contained within observation space 'Box(4,)'. [2016-05-26 03:10:56,877] Observation '[-2.42284413 -1.47464028 -0.00405668 -0.15805834]' is not contained within observation space 'Box(4,)'. [2016-05-26 03:10:56,915] Observation '[-2.42800512 -1.81184147 0.01414837 0.08355538]' is not contained within observation space 'Box(4,)'. [2016-05-26 03:10:56,993] Observation '[-2.42121197 -1.45672808 0.08031289 0.17295786]' is not contained within observation space 'Box(4,)'.

It seems the problem is happening to x now, which has a 2.4 limit.

[rafaelcp]

Also, CartPole should be updated to v1, since these changes are breaking some algorithms which used boundary information for normalization.

[gdb]

From chat: "Artem Yankov @yankov 16:17
anyone ran into this #88 looks like it’s still an issue (have just pulled the latest changes from master)"

[JKCooper2]

It seems to be the same type of issue as the previous one where the observation space bounds are the same as the environment failure point bounds, so when you check the state after the failure point it is outside the observation space limits. It's easy enough to increase the limit but would it be better to add some code that if an observation is given that is outside the bounds in a box environment that it caps it to being equal to the bounded value before it's returned (but after done is calculated). Should the observation bounds be a fixed edge state of the environment that the agent can't break out of?