Policy evaluation for GridWorld

[shangtongzhang]

Policy evaluation for GridWorld

[ShangtongZhang]

Figure 4.1 is only policy evaluation not policy iteration, so we should not change the random policy. And even you do policy iteration I do not see any reason to use a softmax, unless you are working on soft q-learning. https://arxiv.org/abs/1704.06440

[cbrom]

So, by changing the action probabilities I am doing policy iteration, and using softmax is not the right way?

[ShangtongZhang]

yes

[cbrom]

Hey I came to a realization that for it to be a policy iteration it must satisfy this one

1. Initialization
   V (s) ∈ R and π(s) ∈ A(s) arbitrarily for all s ∈ S
2. Policy Evaluation
   Repeat
   ∆ ← 0
   For each s ∈ S:
   v ← V (s)
   V (s) ← s ,r p(s , r |s, π(s)) r + γV (s )
   ∆ ← max(∆, |v − V (s)|)
   until ∆ < θ (a small positive number)
3. Policy Improvement
   policy-stable ← true
   For each s ∈ S:
   old-action ← π(s)
   π(s) ← argmax a s ,r p(s , r |s, a) r + γV (s )
   If old-action = π(s), then policy-stable ← f alse
   If policy-stable, then stop and return V ≈ v ∗ and π ≈ π ∗ ; else go to 2

We should have a separate policy evaluation succeeded by one policy improvement,
but it seems that what I did on my code was value iteration, this one

Repeat
∆ ← 0
For each s ∈ S:
v ← V (s)
V (s) ← max a s ,r p(s , r |s, a) r + γV (s )
∆ ← max(∆, |v − V (s)|)
until ∆ < θ (a small positive number)
Output a deterministic policy, π ≈ π ∗ , such that
π(s) = argmax a s ,r p(s , r |s, a) r + γV (s )

here I used softmax as a probability value of each action.