Error when restoring model to run enjoy.py

[openai]

Error when restoring model to run enjoy.py

[j8lp]

I'm getting this same error too. I'm new to tensorflow, but doesn't this mean that saved.data doesn't match the .checkpoint file?
InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [4] rhs shape= [6] [[Node: save/Assign_3 = Assign[T=DT_FLOAT, _class=["loc:@deepq/q_func/action_value/fully_connected_1/biases"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](deepq/q_func/action_value/fully_connected_1/biases, save/RestoreV2_3/_1)]]

[siemanko]

Which version of Tensorflow are you using?

[j8lp]

1.1.0 version for me

[yenchenlin]

1.1.0 version for me

same here.

[siemanko]

I have a hypothesis, but can you tell me your output of gym.make("Breakout-v4").action_space ?

[yenchenlin]

I got:
Discrete(4)

[j8lp]

same here. What's your hypothesis?

If it helps, here is what I get when I load the saved model and run tf.global_variables(). I'm new to tensorflow, so let me know if there is a better command I can use to debug.

[<tf.Variable 'deepq/eps:0' shape=() dtype=float32_ref>, <tf.Variable 'deepq/q_func/convnet/Conv/weights:0' shape=(8, 8, 4, 32) dtype=float32_ref>, <tf.Variable 'deepq/q_func/convnet/Conv/biases:0' shape=(32,) dtype=float32_ref>, <tf.Variable 'deepq/q_func/convnet/Conv_1/weights:0' shape=(4, 4, 32, 64) dtype=float32_ref>, <tf.Variable 'deepq/q_func/convnet/Conv_1/biases:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'deepq/q_func/convnet/Conv_2/weights:0' shape=(3, 3, 64, 64) dtype=float32_ref>, <tf.Variable 'deepq/q_func/convnet/Conv_2/biases:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'deepq/q_func/state_value/fully_connected/weights:0' shape=(7744, 512) dtype=float32_ref>, <tf.Variable 'deepq/q_func/state_value/fully_connected/biases:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'deepq/q_func/state_value/fully_connected_1/weights:0' shape=(512, 1) dtype=float32_ref>, <tf.Variable 'deepq/q_func/state_value/fully_connected_1/biases:0' shape=(1,) dtype=float32_ref>, <tf.Variable 'deepq/q_func/action_value/fully_connected/weights:0' shape=(7744, 512) dtype=float32_ref>, <tf.Variable 'deepq/q_func/action_value/fully_connected/biases:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'deepq/q_func/action_value/fully_connected_1/weights:0' shape=(512, 6) dtype=float32_ref>, <tf.Variable 'deepq/q_func/action_value/fully_connected_1/biases:0' shape=(6,) dtype=float32_ref>, <tf.Variable 'deepq/target_q_func/convnet/Conv/weights:0' shape=(8, 8, 4, 32) dtype=float32_ref>, <tf.Variable 'deepq/target_q_func/convnet/Conv/biases:0' shape=(32,) dtype=float32_ref>, <tf.Variable 'deepq/target_q_func/convnet/Conv_1/weights:0' shape=(4, 4, 32, 64) dtype=float32_ref>, <tf.Variable 'deepq/target_q_func/convnet/Conv_1/biases:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'deepq/target_q_func/convnet/Conv_2/weights:0' shape=(3, 3, 64, 64) dtype=float32_ref>, <tf.Variable 'deepq/target_q_func/convnet/Conv_2/biases:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'deepq/target_q_func/state_value/fully_connected/weights:0' shape=(7744, 512) dtype=float32_ref>, <tf.Variable 'deepq/target_q_func/state_value/fully_connected/biases:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'deepq/target_q_func/state_value/fully_connected_1/weights:0' shape=(512, 1) dtype=float32_ref>, <tf.Variable 'deepq/target_q_func/state_value/fully_connected_1/biases:0' shape=(1,) dtype=float32_ref>, <tf.Variable 'deepq/target_q_func/action_value/fully_connected/weights:0' shape=(7744, 512) dtype=float32_ref>, <tf.Variable 'deepq/target_q_func/action_value/fully_connected/biases:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'deepq/target_q_func/action_value/fully_connected_1/weights:0' shape=(512, 6) dtype=float32_ref>, <tf.Variable 'deepq/target_q_func/action_value/fully_connected_1/biases:0' shape=(6,) dtype=float32_ref>, <tf.Variable 'deepq_1/beta1_power:0' shape=() dtype=float32_ref>, <tf.Variable 'deepq_1/beta2_power:0' shape=() dtype=float32_ref>, <tf.Variable 'deepq/deepq/q_func/convnet/Conv/weights/Adam:0' shape=(8, 8, 4, 32) dtype=float32_ref>, <tf.Variable 'deepq/deepq/q_func/convnet/Conv/weights/Adam_1:0' shape=(8, 8, 4, 32) dtype=float32_ref>, <tf.Variable 'deepq/deepq/q_func/convnet/Conv/biases/Adam:0' shape=(32,) dtype=float32_ref>, <tf.Variable 'deepq/deepq/q_func/convnet/Conv/biases/Adam_1:0' shape=(32,) dtype=float32_ref>, <tf.Variable 'deepq/deepq/q_func/convnet/Conv_1/weights/Adam:0' shape=(4, 4, 32, 64) dtype=float32_ref>, <tf.Variable 'deepq/deepq/q_func/convnet/Conv_1/weights/Adam_1:0' shape=(4, 4, 32, 64) dtype=float32_ref>, <tf.Variable 'deepq/deepq/q_func/convnet/Conv_1/biases/Adam:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'deepq/deepq/q_func/convnet/Conv_1/biases/Adam_1:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'deepq/deepq/q_func/convnet/Conv_2/weights/Adam:0' shape=(3, 3, 64, 64) dtype=float32_ref>, <tf.Variable 'deepq/deepq/q_func/convnet/Conv_2/weights/Adam_1:0' shape=(3, 3, 64, 64) dtype=float32_ref>, <tf.Variable 'deepq/deepq/q_func/convnet/Conv_2/biases/Adam:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'deepq/deepq/q_func/convnet/Conv_2/biases/Adam_1:0' shape=(64,) dtype=float32_ref>, <tf.Variable 'deepq/deepq/q_func/state_value/fully_connected/weights/Adam:0' shape=(7744, 512) dtype=float32_ref>, <tf.Variable 'deepq/deepq/q_func/state_value/fully_connected/weights/Adam_1:0' shape=(7744, 512) dtype=float32_ref>, <tf.Variable 'deepq/deepq/q_func/state_value/fully_connected/biases/Adam:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'deepq/deepq/q_func/state_value/fully_connected/biases/Adam_1:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'deepq/deepq/q_func/state_value/fully_connected_1/weights/Adam:0' shape=(512, 1) dtype=float32_ref>, <tf.Variable 'deepq/deepq/q_func/state_value/fully_connected_1/weights/Adam_1:0' shape=(512, 1) dtype=float32_ref>, <tf.Variable 'deepq/deepq/q_func/state_value/fully_connected_1/biases/Adam:0' shape=(1,) dtype=float32_ref>, <tf.Variable 'deepq/deepq/q_func/state_value/fully_connected_1/biases/Adam_1:0' shape=(1,) dtype=float32_ref>, <tf.Variable 'deepq/deepq/q_func/action_value/fully_connected/weights/Adam:0' shape=(7744, 512) dtype=float32_ref>, <tf.Variable 'deepq/deepq/q_func/action_value/fully_connected/weights/Adam_1:0' shape=(7744, 512) dtype=float32_ref>, <tf.Variable 'deepq/deepq/q_func/action_value/fully_connected/biases/Adam:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'deepq/deepq/q_func/action_value/fully_connected/biases/Adam_1:0' shape=(512,) dtype=float32_ref>, <tf.Variable 'deepq/deepq/q_func/action_value/fully_connected_1/weights/Adam:0' shape=(512, 6) dtype=float32_ref>, <tf.Variable 'deepq/deepq/q_func/action_value/fully_connected_1/weights/Adam_1:0' shape=(512, 6) dtype=float32_ref>, <tf.Variable 'deepq/deepq/q_func/action_value/fully_connected_1/biases/Adam:0' shape=(6,) dtype=float32_ref>, <tf.Variable 'deepq/deepq/q_func/action_value/fully_connected_1/biases/Adam_1:0' shape=(6,) dtype=float32_ref>]

[siemanko]

Ok, that is very weird. For me it is Discrete(6). Potential fix is to update your atari-py. Also your gym version is 0.9.1, right?

[yenchenlin]

Hello @siemanko , I tried both

pip install gym[atari] --upgrade
pip install atari-py --upgrade

but just got Requirement already up-to-date and the action space is still Discrete(4).

[siemanko]

What about pip freeze | egrep '(gym=|atari-py=)'

[yenchenlin]

atari-py==0.1.1
gym==0.9.1

[j8lp]

same here. I'm guessing that we must have the wrong breakout environment installed somehow

[siemanko]

Ok, definitely update atari-py. I am closing the issue, but let us know if it does not work.

[j8lp]

So I tried reinstalling atari-py, and the Breakout environment still only has an actionspace of 4. I'm not sure if there is something wrong the .bin file or the interface. It's kind of a blackbox.

But I noticed that my Pong environment has an actionspace of 6. So if you replace Breakout with Pong, it works:

python -m baselines.deepq.experiments.atari.enjoy --model-dir /tmp/models/mo del-atari-prior-duel-breakout-1 --env Pong --dueling

Is Pong supposed to have an actionspace of 6? If not, then there is probably some incompatibility between my system and atari.

[abhibhav14]

I had atari-py 0.0.18 before, and both Breakout and Pong had action spaces Discrete(6).
Upgrading to atari-py 0.1.1 actually broke this, and now my Breakout action space is Discrete(4).
I am using gym 0.9.1 with python3.
@j8lp Both those envs are supposed to have six actions each, No-op, right, left(or up/down), fire, fire+right and fire+left.

[yenchenlin]

Hello @siemanko ,

can you tell me your atari-py version?

[siemanko]

Yeah, it seems like mine is actually a bit outdated. It's atari-py==0.0.20.

[j8lp]

I see. So the solution is to downgrade to 0.0.20? I'm going to try downgrading my Atari when I get back from work.

[ViktorM]

I've hit the same error:
python -m baselines.deepq.experiments.atari.download_model --blob model-atari-prior-duel-breakout-1 --model-dir /tmp/models
Downloading model-atari-prior-duel-breakout-1 to /tmp/models...
97% (125829120 of 128538210) |################################################## | Elapsed Time: 0:03:28 ETA: 0:00:04
(deep-rl) D:\Stanford\Deep_RL>python -m baselines.deepq.experiments.atari.enjoy --model-dir /tmp/models/model-atari-prior-duel-breakout-1 --env Breakout --dueling
2017-06-15 16:08:42.543058: W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE instructions, but these are available on your machine and could speed up CPU computations.
2017-06-15 16:08:42.543284: W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-15 16:08:42.544578: W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-15 16:08:42.545392: W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-15 16:08:42.547625: W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-15 16:08:42.550167: W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-06-15 16:08:42.550868: W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-15 16:08:42.551448: W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
[2017-06-15 16:08:42,558] Making new env: BreakoutNoFrameskip-v4
Traceback (most recent call last):

File "C:\Users\Viktor\Anaconda3\envs\deep-rl\lib\site-packages\tensorflow\python\client\session.py", line 1039, in _do_call
  return fn(*args)
File "C:\Users\Viktor\Anaconda3\envs\deep-rl\lib\site-packages\tensorflow\python\client\session.py", line 1021, in _run_fn
  status, run_metadata)
File "C:\Users\Viktor\Anaconda3\envs\deep-rl\lib\contextlib.py", line 66, in __exit__
  next(self.gen)
File "C:\Users\Viktor\Anaconda3\envs\deep-rl\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 466, in raise_exception_on_not_ok_status
  pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [512,4] rhs shape= [512,6]
       [[Node: save/Assign_4 = Assign[T=DT_FLOAT, _class=["loc:@deepq/q_func/action_value/fully_connected_1/weights"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/cpu:0"](deepq/q_func/action_value/fully_connected_1/weights, save/RestoreV2_4)]]

So what Atari version is required?

[ViktorM]

My is atari-py==0.1.1

[j8lp]

I think the baseline breakout model was compiled against an older version of atari-py where the breakout environment had an action space of size 6. But the problem is that the breakout environment in the current version of atari-py has an action space of size 4. Try running with the pong environment instead of the breakout environment

…

Sent from my iPhone

On Jun 15, 2017, at 4:22 PM, ViktorM ***@***.***> wrote: My is atari-py==0.1.1 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[ViktorM]

Yep, pong environment download and play are working for me.

[DanielTakeshi]

Yes, this works for me, downgrading to atari-py 0.0.20. This should probably go in the setup.py file.

[BNSneha]

Neither atari 0.0.20 nor 0.1.1 gets rid of the tensor dimensions mismatch error. What else could be wrong?