yb-guo/sac

Implementation of Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor.

SAC

Implementation of Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

Added another branch for Soft Actor-Critic Algorithms and Applications -> SAC_V1.

Soft Q-Learning uses the following objective function instead of the conventional expected cumulative return:

The entropy term is also maximized which have two major benefits:

1. The exploration will be intelligently tuned and maximized as much as need, so the exploration/exploitation trade off is well satisfied.
2. It prevent the learning procedure to get stuck in a local optima which results to a suboptimal policy.

Demos

Humanoid-v2	Walker2d-v2	Hopper-v2

Results

Humanoid-v2	Walker2d-v2	Hopper-v2

Dependencies

• gym == 0.17.2
• mujoco-py == 2.0.2.13
• numpy == 1.19.1
• psutil == 5.4.2
• torch == 1.4.0

Installation

pip3 install -r requirements.txt

Usage

python3 main.py

• You may use Train flag to specify whether to train your agent when it is True or test it when the flag is False.
• There are some pre-trained weights in pre-trained models dir, you can test the agent by using them; put them on the root folder of the project and turn Train flag to False.

Environment tested

• Humanoid-v2
• Hopper-v2
• Walker2d-v2
• HalfCheetah-v2

Reference

1. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, Haarnoja et al., 2018
2. Soft Actor-Critic Algorithms and Applications, Haarnoja et al., 2018

Acknowledgement

All credits goes to @pranz24 for his brilliant Pytorch implementation of SAC.
Special thanks to @p-christ for SAC.py