oxwhirl/treeqn

treeqn's Issues

Reward loss targets don't account for episodes that finish within n steps

Hi,

Great work. I enjoyed reading the paper and I replicated your work independently.

I noticed minor performance difference between the two implementations and I just noticed that while computing the target for reward loss, you aren't accounting for the episodes that finished within nsteps (whereas for q loss it is being correctly accounted for).

Specifically, proc_seq.append(seq[env, t+offset:t+offset+depth, :]) in line 32 of treeqn_utils.py doesn't check for done flags.

If the episode finished at time t=3, this error will make the target for d=1 at t=3 to be reward at t=4, which is wrong. Can you please clarify if my understanding is correct?

RecursionError: maximum recursion depth exceeded

run python treeqn/nstep_run.py, got the following RecursionError,

File "/home/ubuntu/projects/treeqn/treeqn/utils/pytorch_utils.py", line 67, in ortho_init
   ortho_init(tensor.data, scale=scale)
 File "/home/ubuntu/projects/treeqn/treeqn/utils/pytorch_utils.py", line 67, in ortho_init
   ortho_init(tensor.data, scale=scale)
 File "/home/ubuntu/projects/treeqn/treeqn/utils/pytorch_utils.py", line 67, in ortho_init
   ortho_init(tensor.data, scale=scale)
 File "/home/ubuntu/projects/treeqn/treeqn/utils/pytorch_utils.py", line 66, in ortho_init
   if isinstance(tensor, Variable):
RecursionError: maximum recursion depth exceeded

Then I dig more, found:

ERROR - treeqn_log.treeqn - Failed after 0:00:03!
Traceback (most recent calls WITHOUT Sacred internals):
  File "/home/ubuntu/anaconda3/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "treeqn/nstep_run.py", line 323, in main
    train()
  File "treeqn/nstep_run.py", line 198, in train
    model = create_model(env)
  File "treeqn/nstep_run.py", line 163, in create_model
    model = policy(env.observation_space, env.action_space, env.num_envs, nsteps, nstack)
  File "/home/ubuntu/projects/treeqn/treeqn/models/models.py", line 63, in __init__
    encoder = atari_encoder(ob_shape[1])
  File "/home/ubuntu/projects/treeqn/treeqn/models/encoding.py", line 8, in atari_encoder
    nn_init(nn.Conv2d(in_channels, 16, kernel_size=8, stride=4), w_scale=np.sqrt(2)),
  File "/home/ubuntu/projects/treeqn/treeqn/utils/pytorch_utils.py", line 87, in nn_init
    w_init(module.weight, w_scale)

Any suggestion?