Comments (4)
@ku2482
Hi! I used the entropy loss, and code just runs well. And I still have a question about the returns. Could you please tell me why the train return is so far from the test return. I checked train_episode() and evaluate() in base_agent.py, but I couldn't get the explaination here.
from fqf-iqn-qrdqn.pytorch.
In a personal communication, the author mentions the following. Your problem seems to be related to it.
We did not use the regularization term in our implementation. The reason why we mention this term is because that in some rare case (about 1 out of 20 seeds), the proposed fraction may degenerates into a deterministic one (e.g. [0,0,0,...,1,0,...]).
In FQF paper, the author proposed to use the entropy bonus (regularization term) to prevent it.
I implemented it so that you can use the entropy bonus setting ent_coef
bigger than 0 in the config.
(I haven't tested it due to the limited resources.)
Thanks.
from fqf-iqn-qrdqn.pytorch.
@ku2482
Thank you! I'll do experiments immediately, and see if there is any change. I've done with seed 0 and seed 5, but they all fall into the same situation eventually. Hope a regularizer could help!
from fqf-iqn-qrdqn.pytorch.
@Bowen-He
Thank you for sharing the result ;)
We clip the reward when training.
https://github.com/ku2482/fqf-iqn-qrdqn.pytorch/blob/master/fqf_iqn_qrdqn/env.py#L166
from fqf-iqn-qrdqn.pytorch.
Related Issues (17)
- Question on `update_interval` argument HOT 2
- Question with running the code HOT 8
- A function about continuing training HOT 2
- calculation of loss HOT 2
- Convolution issue with Dimensions in states/embedded_states HOT 1
- Question on QR-DQN calculate_quantile_huber_loss HOT 1
- Element-wise product HOT 1
- Incorrect Q-Value calculation in "qrdqn" agent HOT 2
- fraction proposal network of FQF HOT 1
- A question for Fraction Proposal Network in FQF HOT 2
- one of the variables needed for gradient computation has been modified by an inplace operation HOT 6
- No performance in all three algorithms HOT 6
- Could you please help me with the proof for proposition 1? HOT 7
- Could you please tell me how it to run 'BreakoutDeterministic-v4'? HOT 4
- Questions of several implementation details HOT 2
- Questions of several implementation details HOT 10
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fqf-iqn-qrdqn.pytorch.