Minimal implementation of Stochastic Policy Gradient Algorithm in Keras
This PG agent seems to get more frequent wins after about 8000 episodes. Below is the score graph.
Minimal Monte Carlo Policy Gradient (REINFORCE) Algorithm Implementation in Keras
License: MIT License
I think the code in the agent training process
Line 56 in b83f050
Hey mate,
Great work, but I think you normalization of the discounted rewards is wrong.
pg.py_line 64: rewards = rewards / np.std(rewards - np.mean(rewards))
should maybe be:
rewards = (rewards - np.mean(rewards)) / np.std(rewards - np.mean(rewards))
I do understand the backpropagation in policy gradient networks, but am not sure how your code work keras's auto-differentiation.
That is, how you transform it into a supervised learning problem.
For example, the code below:
Y = self.probs + self.learning_rate * np.squeeze(np.vstack([gradients]))
Why is Y not 1-hot vector for the action taken?
You compute the gradient assuming the action is correct, Y is one-hot vector. Then you multiplies it by the reward in the corresponding time-step. But while training you feed it as the correction.
I think one could multiply the rewards by one-hot vector instead. And then feed it straight away.
If possible please clarify my doubt. :)
https://github.com/keon/policy-gradient/blob/master/pg.py#L67
Thanks.
prob = aprob / np.sum(aprob)
https://github.com/keon/policy-gradient/blob/master/pg.py#L46
I am not sure if this line is really required, as they would be already normalized due to softmax. Please let me know in case I am missing something.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.