ying-wen/malib_deprecated

A Multi-agent Learning Framework

---

This repository has been deprecated, please find new MALib repository at https://github.com/sjtu-marl/malib.

Multi-Agent Reinforcement Learning Framework

This Framework aims to provide an easy to use toolkit for Multi-Agent Reinforcement Learning research. Overall architecture:

Environment: There are two differences for Multi-Agent Env Class: 1. The step(action_n) accepts n actions at each time; 2. The Env class needs a MAEnvSpec property which describes the action spaces and observation spaces for all agents.

Agent: the agent class has no difference than common RL agent, it uses the MAEnvSpec from Env Class to init the policy/value nets and replay buffer.

MASampler: Because the agents have to rollout simultaneously, a MASampler Class is designed to perform the sampling steps and add/return the step tuple to each agent's replay buffer.

MATrainer: In single agent, the trainer is included in the Agent Class. However, due to the complexity of Multi-Agent Training, which has to support independent/centralized/communication/opponent modelling, it is necessary to have a MATrainer Class to abstract these requirements from Agent Class. This is the core for Multi-agent training.

Installation

Required Python Version: >= 3.6

• Using Local Python Environment:

cd malib
sudo pip3 install -r requirements.txt
sudo pip3 install -e .

• Using virtualenv Environment:

cd malib
python3 -m venv env
source env/bin/activate
pip3 install -r requirements.txt
pip3 install -e .

• Using Conda Environment:

cd malib
conda env create --file=environment.yml
conda activate malib
conda develop ./

or

cd malib
conda env create -n malib python=3.7
conda activate malib
pip install -r requirements.txt
conda develop ./

Run Example

cd examples
python run_trainer.py

Testing Code

python -m pytest tests

Testing With Keyword

python -m pytest tests -k "environments"

Reference Projects

The project implementation has referred much and adopted some codes from the following projects: agents, maddpg, softlearning, garage, markov-game, multiagent-particle-env. Thanks a lot!

malib_deprecated's Issues

Twin Delayed Deep Deterministic

https://arxiv.org/abs/1802.09477

Base Environment

Currently, most, if not all, the environment are independent. It might be useful to include the base environment, which will define the interface of all environments. This can be useful. For example

class DifferentialGame:
    def __init__(self, game_name, agent_num, action_range=(-10, 10)):
        self.game = game_name
        self.agent_num = agent_num
        self.action_range = action_range
        ....
    ....

Can inherit BaseGame , which is an environment interface

Soft Actor-Critic

https://arxiv.org/abs/1812.05905

DQN with Distribution RL

https://arxiv.org/abs/1707.06887

Environment Useful Error + Tests

For most of the environments, to check for the configuration, there are mostly an assert statements, which checks whether we can create environment out of specified configuration.

It would be better to have explicit error/exception signal, rather than assert, so that it would be more useful for users. The example change would be

if self.agent_num == 2:
    raise WrongAgentNumberException("The agent number for game: penalty, should be 2")

And normally, there should be a test associated with it

ModuleNotFoundError: No module named 'gym_hiway'

When I run run run_smarts.py, I get the following error
How can I fix this?

Traceback (most recent call last):
  File "examples/run_smarts.py", line 16, in <module>
    from gym_hiway.env.list_hiway_env import ListHiWayEnv
ModuleNotFoundError: No module named 'gym_hiway'

https://github.com/ying-wen/malib/blob/7cd2a4af55cf1f56da8854e26ea7a4f3782ceea2/examples/run_smarts.py#L16

Environment Specification

For example in the class MatrixGame the base-code asserts the number of agent_num and action_num. Why would the user has to specify these number when it is already predefined (as an assertion)

Example,

if self.game == 'coordination_0_0':
    assert self.agent_num == 2
    assert self.action_num == 2
    self.payoff[0]=[[1,-1],
                   [-1,-1]]
    self.payoff[1]=[[1,-1],
                   [-1,-1]]

A2C Implementation

PPO Implementation

In order to gain more understanding of the interface of the repo, we should try to implement PPO from scratch https://arxiv.org/abs/1707.06347

UNREAL Agent

https://arxiv.org/abs/1611.05397

Environment Logger (Possibly Filterable Logger)

Is there any available/useful to have logger for environment (For Debugging Purposes) since currently, we only have print statement to log the output

def step(self, actions):
    assert len(actions) == self.agent_num
    print('actions', actions)
    actions = np.array(actions).reshape((self.agent_num,)) * self.action_range[1]
    print('scaled', actions)
    reward_n = np.zeros((self.agent_num,))
    ...

It might be useful to have a logger that is able to log in certain level specified by the user based on verbosity(verbose/normal/minimal) or based on user filter (environment/model)

DDPG

https://arxiv.org/abs/1509.02971 and can be used for MADDPG

R2D2

https://openreview.net/forum?id=r1lyTjAqYX

TRPO

https://arxiv.org/abs/1502.05477

Multi-Agent Trainer Abstraction

Design the interfaces for Multi-Agent Trainer Abstraction, which controls the training flow of multi-agent learning.
Add MADDPG as an example.

DQN Rainbow

https://arxiv.org/abs/1710.02298

run run_trainer.py

When I run run_trainer.py, it occurs that:
Exception has occurred: AttributeError
module 'tensorflow' has no attribute 'function'
File "E:\google\malib-master\malib\agents\ddpg\maddpg.py", line 194, in MADDPGAgent
@tf.function
File "E:\google\malib-master\malib\agents\ddpg\maddpg.py", line 11, in
class MADDPGAgent(OffPolicyAgent):
File "E:\google\malib-master\malib\agents_init_.py", line 1, in
from malib.agents.ddpg.maddpg import MADDPGAgent
File "E:\googlemalib-master\examples\run_trainer.py", line 4, in
from malib.agents.agent_factory import *

New project Support SMARTS?

Hi! If the new project support SMARTS in https://github.com/sjtu-marl/malib?

DQN and its Variants

DQN with Noisy Networks

https://arxiv.org/abs/1706.10295

DQN With Prioritized replay

https://arxiv.org/abs/1511.05952

policy serialize/deserialize issues

policy: test_serialize_deserialize is not passed in tensorflow==2.0.0(rc0,rc1).

>       serialized = pickle.dumps(self.policy)
E       TypeError: can't pickle _thread._local objects

Hyperparameter List Page

Having a dedicate page for list of hyper-parameters, for all of the experiment/benchmark. This can be very helpful for other people.

Environment wrapper for third-party Gym/Ray compatible multi-agent environment

Making the Particle Environment under the current environment abstraction. Instead of gym.Env