Question about running FQF agent on Breakout

[toshikwa]

Question about running FQF agent on Breakout

[Bowen-He]

@ku2482
Hi! I used the entropy loss, and code just runs well. And I still have a question about the returns. Could you please tell me why the train return is so far from the test return. I checked train_episode() and evaluate() in base_agent.py, but I couldn't get the explaination here.

[toshikwa]

@Bowen-He

In a personal communication, the author mentions the following. Your problem seems to be related to it.

We did not use the regularization term in our implementation. The reason why we mention this term is because that in some rare case (about 1 out of 20 seeds), the proposed fraction may degenerates into a deterministic one (e.g. [0,0,0,...,1,0,...]).

In FQF paper, the author proposed to use the entropy bonus (regularization term) to prevent it.
I implemented it so that you can use the entropy bonus setting ent_coef bigger than 0 in the config.
(I haven't tested it due to the limited resources.)

Thanks.

[Bowen-He]

@ku2482
Thank you! I'll do experiments immediately, and see if there is any change. I've done with seed 0 and seed 5, but they all fall into the same situation eventually. Hope a regularizer could help!

[toshikwa]

@Bowen-He
Thank you for sharing the result ;)

We clip the reward when training.
https://github.com/ku2482/fqf-iqn-qrdqn.pytorch/blob/master/fqf_iqn_qrdqn/env.py#L166