miffyli/nle-sample-factory-baseline

nle-sample-factory-baseline's Issues

Pretrained checkpoint and configuration for reaching the described reward.

Thanks so much for sharing this implementation, I think it's a really fantastic implementation. I would love to try it out and see if I can duplicate the results but I'm running into some issues.

When I run ./train_baseline.sh I see messages like this:
Discarding 130 old rollouts, cut by policy lag threshold 10000 (learner 0)
and the learner doesn't seem to learn anything. I know this can be fixed by setting the --max_policy_lag but the fact it's coming up now implies that either my local setup is different (maybe versions?) or a different configuration was used? If it's the latter would be possible to share a working configuration?

I tried training from scratch by removing the train_dir which increased the episode_len without marginal increase to the reward before running into the same "discarding old rollouts" issue.
Discarding 130 old rollouts, cut by policy lag threshold 10000 (learner 0)

Similarly when using the run.sh script the reward seems to be a lot less than what is described. Perhaps the checkpoint doesn't contain a fully trained policy?

[2022-02-24 21:05:07,513][06642] Episode finished for agent 0 at 49393 frames. Reward: 34.000, true_reward: 34.000
[2022-02-24 21:05:07,513][06642] Avg episode rewards: #0: 41.200, true rewards: #0: 41.200
[2022-02-24 21:05:07,514][06642] Avg episode reward: 41.200, avg true_reward: 41.200