Chapter 5: Monte Carlo ES initial policy

[jerome-white]

Chapter 5: Monte Carlo ES initial policy

[ShangtongZhang]

It seems I did't notice this sentence when I implemented this so I just used a random one.

As the initial policy we use the policy evaluated in the previous blackjack example, that which sticks only on 20 or 21.

I have no idea how exactly you use targetPolicyPlayer as the initial policy so I can't give further advise. But I will also try it myself in the future.

[jerome-white]

230c230
<         initialAction = np.random.choice(actions)
---
>         initialAction = int(targetPolicyPlayer(*initialState))

[ShangtongZhang]

This won't work. By initial policy it means the initial policy for the whole training process, not the initial policy for each episode.

[jerome-white]

Another interpretation would be to use targetPolicyPlayer once for each state, the first time that state has been explored. In this case, the policy array could be viewed as a set: if the state is not in the set, use targetPolicyPlayer to decide the action; use behaviorPolicy otherwise.

However, in my personal implementation I wasn't able to get this logic to work either. I have the feeling that random is the correct thing to do; I just wish I knew what the authors had in mind when they wrote what they did in the exercise.

[ShangtongZhang]

https://github.com/ShangtongZhang/reinforcement-learning-an-introduction/blob/latest/chapter05/blackjack.py#L230

This line may explain how it works