chintantrivedi/rl-bot-football

An RL agent for the Google Football environment

rl-bot-football's Issues

getting 0 rewards for all episodes

Hello

Thanks for the detailed implementation.. but I get 0 rewards. And consequently, no model is saved.

Which tensorflow/keras version are you using?

When calling the fit function on the actor model, Im getting the following error when executing your complete code (with image_based = True):

ValueError: Data cardinality is ambiguous:
x sizes: 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 128, 128, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
y sizes: 128
Please provide data which shares the same first dimension.

Eager execution function cannot be Keras symbolic tensors

Hi, @ChintanTrivedi I am using the modified version of your code to train the environment created using the Unity engine.

[I have modified the code to handle this].
Action space = Continuous
Observation space = Vectorized

TensorFlow version = 2.3.0
Python version = 3.8.3

I think the error is introduced due to the custom loss function call. [Tensor is passed to the loss function so eager execution is giving error]

So while I am trying to fit the model to the collected data, I am getting the following errors.
So, please can you tell me what am I doing wrong?

Epoch 1/8
Traceback (most recent call last):
  File "C:\Users\dhyey\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\eager\execute.py", line 59, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
TypeError: An op outside of the function building code is being passed
a "Graph" tensor. It is possible to have Graph tensors
leak out of the function building context by including a
tf.init_scope in your function building code.
For example, the following function will fail:
  @tf.function
  def has_init_scope():
    my_constant = tf.constant(1.)
    with tf.init_scope():
      added = my_constant * 2
The graph tensor has name: old_prediction:0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "temp.py", line 259, in <module>
    agent.train()
  File "temp.py", line 228, in train
    actor_loss = self.actor.fit(
  File "C:\Users\dhyey\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\keras\engine\training.py", line 108, in _method_wrapper
    return method(self, *args, **kwargs)
  File "C:\Users\dhyey\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1098, in fit
    tmp_logs = train_function(iterator)
  File "C:\Users\dhyey\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\eager\def_function.py", line 780, in __call__
    result = self._call(*args, **kwds)
  File "C:\Users\dhyey\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\eager\def_function.py", line 840, in _call
    return self._stateless_fn(*args, **kwds)
  File "C:\Users\dhyey\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\eager\function.py", line 2829, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "C:\Users\dhyey\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\eager\function.py", line 1843, in _filtered_call
    return self._call_flat(
  File "C:\Users\dhyey\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\eager\function.py", line 1923, in _call_flat
    return self._build_call_outputs(self._inference_function.call(
  File "C:\Users\dhyey\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\eager\function.py", line 545, in call
    outputs = execute.execute(
  File "C:\Users\dhyey\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\eager\execute.py", line 72, in quick_execute
    raise core._SymbolicException(
tensorflow.python.eager.core._SymbolicException: Inputs to eager execution function cannot be Keras symbolic tensors, but found [<tf.Tensor 'old_prediction:0' shape=(None, 2) dtype=float32>, <tf.Tensor 'advantage:0' shape=(None, 1) dtype=float32>, <tf.Tensor 'reward:0' shape=(None, 1) dtype=float32>, <tf.Tensor 'value:0' shape=(None, 1) dtype=float32>]

Here is the code which is calling the .fit() method to the model:

while not target_reached and iters < max_iters:
            states = []
            actions = []
            old_predictions = []
            rewards = []
            values = []
            masks = []
            state = None
            self.env.reset()
            step_result = self.env.get_steps(self.behavior_name)
            state = step_result[0].obs[0]  # observation

            for itr in range(ppo_steps):
                #generate the actions using predict method on actor model.
                action, action_matrix, predicted_action = self.get_action(state, True) 

                q_value = self.critic.predict(state, steps=1)

                next_state, reward, done = self.step(action) # apply the actions to the env and returns the env response. 

                print('itr: ' + str(itr) + ', reward=' + str(reward) + ', q_val=' + str(q_value))
                mask = not done

                states.append(state)
                actions.append(action_matrix)
                old_predictions.append(predicted_action)
                rewards.append(reward)
                values.append(q_value)
                masks.append(mask)

                state = next_state
                if done:
                    self.env.reset()

            q_value = self.critic.predict(state, steps=1)
            values.append(q_value)
            returns, advantages = self.get_advantages(values, masks, rewards)
            
            # reshaping
            states = np.reshape(states, (len(states), self.state_dims))
            actions = np.reshape(actions, (len(actions), self.n_actions))
            old_predictions = np.reshape(old_predictions, (len(old_predictions), self.n_actions))

            rewards = np.reshape(rewards, (-1, 1))
            values = np.reshape(values, (len(values), 1))
            advantages = np.reshape(advantages, (len(advantages), 1))
            returns = np.reshape(returns, (len(returns), 1))

            actor_loss = self.actor.fit(
                [states, old_predictions, advantages, rewards, values[:-1]],
                [actions],
                verbose=True, shuffle=True, epochs=8,
                callbacks=[self.tensor_board]
            )
            critic_loss = self.critic.fit(
                [states], [returns],
                shuffle=True, epochs=8, verbose=True,
                callbacks=[self.tensor_board]
            )