Comments (2)
看来你自己已经明白了,reinforce的更新是在跑完整个序列后,ac的不是,代码如下,具体看step在哪里运行
for i in reversed(range(len(reward_list))): # 从最后一步算起
reward = reward_list[i]
state = torch.tensor([state_list[i]], dtype=torch.float).to(self.device)
action = torch.tensor([action_list[i]]).view(-1, 1).to(self.device)
log_prob = torch.log(self.policy_net(state).gather(1, action))
G = self.gamma * G + reward
loss = -log_prob * G # 每一步的损失函数
loss.backward() # 反向传播计算梯度
self.optimizer.step() # 梯度下降
# 时序差分目标
td_target = rewards + self.gamma * self.critic(next_states) * (1 - dones)
td_delta = td_target - self.critic(states) # 时序差分误差
log_probs = torch.log(self.actor(states).gather(1, actions))
actor_loss = torch.mean(-log_probs * td_delta.detach())
# 均方误差损失函数
critic_loss = torch.mean(F.mse_loss(self.critic(states), td_target.detach()))
self.actor_optimizer.zero_grad()
self.critic_optimizer.zero_grad()
actor_loss.backward() # 计算策略网络的梯度
critic_loss.backward() # 计算价值网络的梯度
self.actor_optimizer.step() # 更新策略网络的参数
self.critic_optimizer.step() # 更新价值网络的参数
from hands-on-rl.
看来你自己已经明白了,reinforce的更新是在跑完整个序列后,ac的不是,代码如下,具体看step在哪里运行
for i in reversed(range(len(reward_list))): # 从最后一步算起 reward = reward_list[i] state = torch.tensor([state_list[i]], dtype=torch.float).to(self.device) action = torch.tensor([action_list[i]]).view(-1, 1).to(self.device) log_prob = torch.log(self.policy_net(state).gather(1, action)) G = self.gamma * G + reward loss = -log_prob * G # 每一步的损失函数 loss.backward() # 反向传播计算梯度 self.optimizer.step() # 梯度下降# 时序差分目标 td_target = rewards + self.gamma * self.critic(next_states) * (1 - dones) td_delta = td_target - self.critic(states) # 时序差分误差 log_probs = torch.log(self.actor(states).gather(1, actions)) actor_loss = torch.mean(-log_probs * td_delta.detach()) # 均方误差损失函数 critic_loss = torch.mean(F.mse_loss(self.critic(states), td_target.detach())) self.actor_optimizer.zero_grad() self.critic_optimizer.zero_grad() actor_loss.backward() # 计算策略网络的梯度 critic_loss.backward() # 计算价值网络的梯度 self.actor_optimizer.step() # 更新策略网络的参数 self.critic_optimizer.step() # 更新价值网络的参数
请问可以再详细的介绍一下吗?我还是不太理解,示例代码中AC算法执行的是 rl_utils.py 中的 train_on_policy_agent 函数,是收集一整个 Episode 之后的信息再执行 AC 的 update 函数,即使 AC 的 update 函数不是跟 REINFORCE 算法中的 update 函数一样用for循环,但是也不符合每个时间步更新策略这个特点。
from hands-on-rl.
Related Issues (20)
- 21章MADDPG代码问题,存在维度不匹配 HOT 2
- 运行环境 HOT 1
- SAC算法——状态价值函数存在问题
- 7.4 DQN 算法反向传播有没有进行求导?? HOT 1
- 公式错误---https://hrl.boyuai.com/chapter/1/时序差分算法#55-q-learning-算法 HOT 1
- 将AC改为off-policy后,每次训练500条左右的经验就会报错,显示action_dist = torch.distributions.Categorical(probs)这行代码的运行结果为tensor([[nan, nan]] HOT 3
- 用spyder跑PPO代码,kernel自动关闭了 HOT 1
- DQN ReplayBuffer HOT 1
- SAC伪代码存在一点小问题
- PPO在单摆实验中为什么要对reward=(reward+8)/8的修改呢? HOT 2
- 网页版教程 3.3.2 价值函数 推导过程有些模糊
- CartPole-v0环境训练reward超过上限值200? HOT 1
- 关于环境初始化的一点提示
- 多臂老虎机的代码里面init_prob为什么是1.0?
- 第十三章 DDPG算法 代码实践中的一点疏漏
- 关于web教程布局的建议
- 马尔可夫决策过程,MDP转化为MRP时计算的P疑似有误 HOT 1
- https://www.boyuai.com/进不去了
- 第九章策略梯度的损失函数 HOT 1
- 3.6. 占用度量,为何逆序计算? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hands-on-rl.