chapter5

[shangtongzhang]

chapter5

[tinglo]

Thank you for your reply.
But the Line 137 appends the state before it changes.
I output every states in an episode:
(the first element is action, the second element is state)

And I output the value of the state (False, 14, 4), it shows 0.66666667,
since the count of every states initialize 1, and state (False, 14, 4) is counted twice-> (1+1)/(1+1+1) = 0.66666667
But the state(False, 14, 4) shouldn't be counted twice.

I'm sorry for asking too much.
I will introduce this chapter in a class, so I want to understand the deatails.
Thank you very much.

[ShangtongZhang]

Line 137 won't append the initial state. It just appends (action, next_state).

[ShangtongZhang]

Oh it's indeed a bug, thanks for pointing this out! I will fix it soon.

[ShangtongZhang]

I just pushed the fix, thanks again! It will be much appreciated if you let me know once you find new bugs.

[tinglo]

Sure!
My friends and I are studying RL recently, and your code really helps us understand more clearly!
We're really grateful for these implementations!