sisl/madrl

Repo containing code for multi-agent deep reinforcement learning (MADRL).

MADRL

Note: A maintained version of the first three environments with various fixes is included with PettingZoo (https://github.com/PettingZoo-Team/PettingZoo, https://pettingzoo.farama.org/environments/sisl/)

This package provides implementations of the following multi-agent reinforcement learning environemnts:

• Pursuit Evastion
• Waterworld
• Multi-Agent Walker
• Multi-Ant

Requirements

This package requires both OpenAI Gym and a forked version of rllab (the multiagent branch). There are a number of other requirements which can be found in rllab/environment.yml file if using anaconda distribution.

Setup

The easiest way to install MADRL and its dependencies is to perform a recursive clone of this repository.

git clone --recursive [email protected]:sisl/MADRL.git

Then, add directories to PYTHONPATH

export PYTHONPATH=$(pwd):$(pwd)/rltools:$(pwd)/rllab:$PYTHONPATH

Install the required dependencies. Good idea is to look into rllab/environment.yml file if using anaconda distribution.

Usage

Example run with curriculum:

python3 runners/run_multiwalker.py rllab \ # Use rllab for training
    --control decentralized \ # Decentralized training protocol
    --policy_hidden 100,50,25 \ # Set MLP policy hidden layer sizes
    --n_iter 200 \ # Number of iterations
    --n_walkers 2 \ # Starting number of walkers
    --batch_size 24000 \ # Number of rollout waypoints
    --curriculum lessons/multiwalker/env.yaml

Details

Policy definitions exist in rllab/sandbox/rocky/tf/policies.

Citation

Please cite the accompanied paper, if you find this useful:

@inproceedings{gupta2017cooperative,
  title={Cooperative multi-agent control using deep reinforcement learning},
  author={Gupta, Jayesh K and Egorov, Maxim and Kochenderfer, Mykel},
  booktitle={International Conference on Autonomous Agents and Multiagent Systems},
  pages={66--83},
  year={2017},
  organization={Springer}
}

madrl's Issues

ode problem

I used pip install ode to install ode in Python 3. However, in box_carrying.py, line 88,

        self._odebody = ode.Body(world)

There is no Body in ode. How can I install ode correctly in Python 3.

Shape error of tensorflow

I faced this error:
python3 runners/run_multiwalker.py rllab \--control decentralized \--policy_hidden 100,50,25 \--n_iter 200 \--n_walkers 2 \--batch_size 24000 \--curriculum lessons/multiwalker/env.yaml [2019-07-09 16:30:31,498] Using NumPy C-API based implementation for BLAS functions. [2019-07-09 16:30:32,017] From /home/zkg/.local/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. Traceback (most recent call last): File "runners/run_multiwalker.py", line 58, in <module> main(RunnerParser(ENV_OPTIONS)) File "runners/run_multiwalker.py", line 52, in main run(curr) File "/home/zkg/MADRL/runners/rurllab.py", line 355, in __call__ env, policy = rllab_envpolicy_parser(self.env, self.args) File "/home/zkg/MADRL/runners/rurllab.py", line 175, in rllab_envpolicy_parser min_std=args.min_std, name='policy') File "/home/zkg/MADRL/rllab/sandbox/rocky/tf/policies/gaussian_mlp_policy.py", line 133, in __init__ dist_info_sym = self.dist_info_sym(mean_network.input_layer.input_var, dict()) File "/home/zkg/MADRL/rllab/sandbox/rocky/tf/policies/gaussian_mlp_policy.py", line 147, in dist_info_sym mean_var, std_param_var = L.get_output([self._l_mean, self._l_std_param], obs_var) File "/home/zkg/MADRL/rllab/sandbox/rocky/tf/core/layers.py", line 1792, in get_output all_outputs[layer] = layer.get_output_for(layer_inputs, **kwargs) File "/home/zkg/MADRL/rllab/sandbox/rocky/tf/core/layers.py", line 354, in get_output_for tile_arg = tf.concat(0, [tf.shape(input)[:ndim - 1], [1]]) File "/home/zkg/.local/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py", line 180, in wrapper return target(*args, **kwargs) File "/home/zkg/.local/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 1254, in concat tensor_shape.scalar()) File "/home/zkg/.local/lib/python3.6/site-packages/tensorflow/python/framework/tensor_shape.py", line 1023, in assert_is_compatible_with raise ValueError("Shapes %s and %s are incompatible" % (self, other)) ValueError: Shapes (2, 1) and () are incompatible
How can I fix this problem? Which version of tensorflow should I use?
Thanks!!!

Problem in rllab/misc/evaluate.py

After taking a look, I resolved this bug by changing the line "return dict(ret=rets, retstd=retsstd, discret=discrets, path_reward=path_reward, retlist=retlist, **dictinfos)" to "return dict(ret=rets, retstd=retsstd, discret=discrets, path_reward=pr, retlist=retlist, **dictinfos)". If I'm wrong please let me know. Thanks!

Traceback (most recent call last):
File "/home/yantian/research/MADRL/runners/run_multiwalker.py", line 59, in
main(RunnerParser(ENV_OPTIONS))
File "/home/yantian/research/MADRL/runners/run_multiwalker.py", line 53, in main
run(curr)
File "/home/yantian/research/MADRL/runners/rurllab.py", line 360, in call
algo.curriculum_train(curriculum)
File "/home/yantian/research/MADRL/rllab/sandbox/rocky/tf/algos/batch_ma_polopt.py", line 175, in curriculum_train
disc=self.discount)
File "/home/yantian/research/MADRL/rllab/rllab/misc/evaluate.py", line 83, in evaluate
return dict(ret=rets, retstd=retsstd, discret=discrets, path_reward=path_reward, retlist=retlist, **dictinfos)
NameError: global name 'path_reward' is not defined

Pursuit-evasion observation and action spaces

Hello, I have been spending some time now to understand the meaning behind the pursuit-evasion observation matrix but I don't seem to get a clear picture of it. For instance, for a 2 pursuers vs 5 evaders game, I get an observation like
`observation: [array([[[0. , 0. , 0. , 0. ],
[0. , 0. , 0. , 0. ],
[0. , 0. , 0. , 0. ]],

   [[0. , 0. , 0. , 0. ],
    [0. , 0.1, 0. , 0. ],
    [0. , 0. , 0. , 0. ]],

   [[0. , 0. , 0.1, 0. ],
    [0. , 0. , 0. , 0. ],
    [0. , 0. , 0. , 0. ]]]), array([[[0.1, 0. , 0. , 0. ],
    [0. , 0. , 0. , 0. ],
    [0. , 0. , 0. , 0. ]],

   [[0.1, 0. , 0. , 0. ],
    [0. , 0.1, 0. , 0.5],
    [0. , 0. , 0. , 0. ]],

   [[0.1, 0. , 0. , 0. ],
    [0. , 0. , 0. , 0. ],
    [0. , 0. , 0. , 0. ]]])]`

in one iteration. The environment was creating using:
env = PursuitEvade([map_mat], n_evaders=n_evaders, n_pursuers=n_pursuers, obs_range=3, n_catch=2, surround=False, reward_mech='local')
and I have set self.flatten = False to help me view the individual channels. The pursuers are select random actions in this instance.

I wanna know what each channel represents in this case and the meaning of the matrix elements. I have read the paper and it states that each agent recieves a limited-range observation consisting of location of pursuers, evaders, and obstacles. However, I do not know in which order they appear in the experiment and I'm unable to deduce the meaning of the channels from the rendered environment.
Also, I have realized that the 5 possible actions are assigned nominal values (0-4). Please, which value corresponds to which action in this case?
Thanks.

Test pusuit evasion model

I try to test a saved pursuit .pkl model by adding the model path to the argument "check_point" but that doesn't work.
I want to test the model and visualize the behavior of the agents in one complete iteration.
Is there another script in the package that I must use?

decrollout missing from rllab sampler

from rllab.sampler.utils import rollout, decrollout
The decrollout function is missing from the rllab branch sub module. Where can I find the function definition?

install problem

When I tried to install the environment as guide says, namely the command "git clone --recursive [email protected]:sisl/MADRL.git" in Ubunt 16.04, the system reports the error :

Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Could anyone help to solve this problem and tell me how to deal with it? Thanks

How to load the trained parameters which are saved in .pkl?

I use
file=open('/Users/xuehan/MADRL/data/experiment_2017_11_03_15_34_27_439853_CST_7fbba/itr_0.pkl','rb') inf =joblib.load(file) file.close() to load trained parameters but has errors. Hopes you can tell me how to successfully load the trained parameters!
Traceback (most recent call last):
File "run_multiwalker.py", line 60, in
main(RunnerParser(ENV_OPTIONS))
File "run_multiwalker.py", line 54, in main
run(curr)
File "/Users/xuehan/MADRL/runners/rurllab.py", line 395, in call
env, policy = rllab_envpolicy_parser(self.env, self.args)
File "/Users/xuehan/MADRL/runners/rurllab.py", line 185, in rllab_envpolicy_parser
min_std=args.min_std, name='policy')
File "/Users/xuehan/anaconda/envs/rllab3/lib/python3.5/site-packages/sandbox/rocky/tf/policies/gaussian_mlp_policy.py", line 105, in init
inf =joblib.load(file) #读取pkl文件的内容
File "/Users/xuehan/anaconda/envs/rllab3/lib/python3.5/site-packages/joblib/numpy_pickle.py", line 443, in load
with open(filename, 'rb') as file_handle:
TypeError: invalid file: <_io.BufferedReader name='/Users/xuehan/MADRL/data/experiment_2017_11_03_15_34_27_439853_CST_7fbba/itr_0.pkl'>

Error of calling parallel_sampler

Hi,

In ma_sampler.py, the "G.policy.set_param_values(params)" would call a method in parallel_sampler.py, with the line 205: G = _get_scoped_G(G, scope), and the line 7: from rllab.sampler.parallel_sampler import (_get_scoped_G, _worker_set_env_params).

However, I don't see any related method in the parallel_sampler.py. Thank you in advance!

Traceback (most recent call last):
File "/home/yantian/research/MADRL/runners/run_multiwalker.py", line 59, in
main(RunnerParser(ENV_OPTIONS))
File "/home/yantian/research/MADRL/runners/run_multiwalker.py", line 53, in main
run(curr)
File "/home/yantian/research/MADRL/runners/rurllab.py", line 360, in call
algo.curriculum_train(curriculum)
File "/home/yantian/research/MADRL/rllab/sandbox/rocky/tf/algos/batch_ma_polopt.py", line 146, in curriculum_train
paths = self.obtain_samples(itr)
File "/home/yantian/research/MADRL/rllab/sandbox/rocky/tf/algos/batch_ma_polopt.py", line 81, in obtain_samples
return self.sampler.obtain_samples(itr)
File "/home/yantian/research/MADRL/rllab/sandbox/rocky/tf/samplers/batch_ma_sampler.py", line 52, in obtain_samples
scope=self.algo.scope,)
File "/home/yantian/research/MADRL/rllab/rllab/sampler/ma_sampler.py", line 235, in sample_paths
[(policy_params, ma_mode, scope)] * singleton_pool.n_parallel)
File "/home/yantian/research/MADRL/rllab/rllab/sampler/stateful_pool.py", line 83, in run_each
return [runner(self.G, *args_list[0])]
File "/home/yantian/research/MADRL/rllab/rllab/sampler/ma_sampler.py", line 210, in _worker_set_policy_params
G.policy.set_param_values(params)
AttributeError: 'NoneType' object has no attribute 'set_param_values'

How can i change the space in CarRacing-v0 from box to discrete?

I wanna train my agent in CarRacing-v0 environment, but instead of box action/observation spaces I want to use discrete spaces so I can train it with DQN algorithm. there is a sayin in openai-gym that says: "Discreet control is reasonable in this environment as well, on/off discretisation is fine."
https://gym.openai.com/envs/CarRacing-v0/

but I don't know how to do it in code or in pragmatic way.

readme instructions no longer work

I'm getting this after installing the code per the documentation. Something clearly isn't working anymore.

justinkterry@Prophet:~/Documents/MADRL$ python3 runners/run_multiwalker.py rllab --control decentralized --policy_hidden 100,50,25 --n_iter 200 --n_walkers 2 --batch_size 24000 --curriculum lessons/multiwalker/env.yaml
Traceback (most recent call last):
  File "runners/run_multiwalker.py", line 7, in <module>
    from runners import RunnerParser
ModuleNotFoundError: No module named 'runners'

proxy.client("set_state", state_str, async=True) syntax error while running code

When I try to clone the repo using recursive command, I get the following errors:
git clone --recursive [email protected]:sisl/MADRL.git
Cloning into 'MADRL'...
The authenticity of host 'github.com (192.30.253.112)' can't be established.
RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'github.com,192.30.253.112' (RSA) to the list of known hosts.
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Therefore, I cloned only this repo and cloned the rllab and rltools repo manually and added them to the PYTHONPATH as mentioned. I also installed all depenedencies from environment.yml in rllab.

However, when I run the command:
python3 runners/run_multiwalker.py rllab --control decentralized --policy_hidden 100,50,25 --n_iter 200 --n_walkers 2 --batch_size 24000 --curriculum lessons/multiwalker/env.yaml

I get the following errors:
Traceback (most recent call last):
File "runners/run_multiwalker.py", line 10, in
from madrl_environments.walker.multi_walker import MultiWalkerEnv
File "/Users/anirudhajitani/Downloads/Projects/MADRL/madrl_environments/init.py", line 1, in
from rltools.util import EzPickle, stack_dict_list
File "/Users/anirudhajitani/Downloads/Projects/MADRL/rltools/rltools/util.py", line 157
proxy.client("set_state", state_str, async=True)
^
SyntaxError: invalid syntax

I tried to find the the function client, but the prototype for it is completely different. Can somebody tell me what I am foing wrong?

In the pursuit (2vs5), why can't step the actions like [2,2]?

env = pursuit_v3.env(max_cycles=500, x_size=16, y_size=16, local_ratio=1.0, n_evaders=5,
n_pursuers=2,obs_range=7, n_catch=2, freeze_evaders=False, tag_reward=0.01,
catch_reward=5.0, urgency_reward=0.0, surround=True, constraint_window=1.0)

if the joint action is [2,2], how can I step it?
I tried env.step([2,2]), but return AssertionError: action is not in action space

multi_ant.py#L138

MADRL/madrl_environments/mujoco/ant/multi_ant.py

Line 138 in a3651e2

np.square(np.clip(self.model.data.cfrc_ext, -1, 1)))

I think it is not self.mode.data, it is self.data

Issue in installing pip install pettingzoo[sisl] command

When i run the pip install pettingzoo[sisl] command as given on the website https://pettingzoo.farama.org/environments/sisl/
I get the following error while installation. The issue is not with the pip but with box2d-py file.

Building wheels for collected packages: box2d-py
Building wheel for box2d-py (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [16 lines of output]
Using setuptools (version 47.1.0).

Kindly tell the solution

How to install MADRL?

I've downloaded MADRL, but I didn't install it successfully.
Does it need the below command?
conda env create -f environment.yml

ResolvePackageNotFound:

• pygame
• tensorflow=0.10.0rc0

I've installed pygame using "pip intall pygame".

RuntimeError

I tried to run runners>run_multiwalker.py but I'm faced with this error:

File "", line 1, in
runfile('C:/Users/niloo/PycharmProjects/untitled/MADRL-master/runners/run_multiwalker.py', wdir='C:/Users/niloo/PycharmProjects/untitled/MADRL-master/runners')

File "C:\Users\niloo\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 710, in runfile
execfile(filename, namespace)

File "C:\Users\niloo\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 101, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)

File "C:/Users/niloo/PycharmProjects/untitled/MADRL-master/runners/run_multiwalker.py", line 7, in
from runners import RunnerParser

File "", line 961, in _find_and_load

File "", line 946, in _find_and_load_unlocked

File "", line 881, in _find_spec

File "", line 855, in _find_spec_legacy

File "C:\Users\niloo\Anaconda3\lib\site-packages\pyximport\pyximport.py", line 253, in find_module
fp, pathname, (ext,mode,ty) = imp.find_module(fullname,package_path)

File "C:\Users\niloo\Anaconda3\lib\imp.py", line 269, in find_module
raise RuntimeError("'path' must be None or a list, "

RuntimeError: 'path' must be None or a list, not <class '_frozen_importlib_external._NamespacePath'>
What should I do?

rltools is missing

ModuleNotFoundError: No module named 'rltools'

Line 1: MADRL/madrl_environments/init.py

from rltools.util import EzPickle, stack_dict_list

It doesn't work and I didn't find this module here.

make fully observations

In pursuit envs，how can I set up parameters to make agents make fully observations instead of partially observations?

user interface

is there user interface implementation in pursuit evader problem?
in other word UI show me movements of agents.
if yes,how can use it?

Can you tell me how to run the policy in .pkl?

I have load the parameters in .pkl,but when i use it ,tensorflow initialize all the parameter.So that means I started retraining again.How do I use .pkl file to continue training？
I use the code below,but it does not work:
uninit_vars = []
for var in tf.all_variables():
try:
sess.run(var)
except tf.errors.FailedPreconditionError:
uninit_vars.append(var)
init_new_vars_op = tf.initialize_variables(uninit_vars)
sess.run(init_new_vars_op)

rltools is missing

Hi

I just tried to clone the repository but it seems that rltools is gone. Do you know where I can still find it or can you reupload it?

Thanks

ImportError: No module named 'gym.monitoring'

I was using python3.5.4 on MacOS 10.15.4, and gym 0.17.2，but faced with the following problem

Traceback (most recent call last):
  File "/Users/jj/Desktop/MADRL/runners/run_multiwalker.py", line 10, in <module>
    from madrl_environments.walker.multi_walker import MultiWalkerEnv
  File "/Users/jj/Desktop/MADRL/madrl_environments/__init__.py", line 3, in <module>
    from gym.monitoring.video_recorder import ImageEncoder

ImportError: No module named 'gym.monitoring'

some wrong with gym version, but the "rllab/environment.yml" did not mention the gym version, can somebody tell me ?

Train_multi_walker.py doesn't execute

in madrl_environments/walker/train_multi_walker file you declare
' env = MultiWalkerEnv(n_walkers=args.n_walkers) '
and then used env.observation_space but actually MultiWalkerEnv doesn't have an attribute such as observation_space and when i run this file i got following error:

AttributeError: 'MultiWalkerEnv' object has no attribute 'observation_space'

and even after that you use tblog=train_args['tblog'] although GaussianMLPPolicy doesn't have that argument
i will appreciate if you help me and let me know if i am making a mistake ,thank you

proxy.client("set_state", state_str, async=True) syntax error while running code

When I try to clone the repo using recursive command, I get the following errors:
git clone --recursive [email protected]:sisl/MADRL.git
Cloning into 'MADRL'...
The authenticity of host 'github.com (192.30.253.112)' can't be established.
RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'github.com,192.30.253.112' (RSA) to the list of known hosts.
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Therefore, I cloned only this repo and cloned the rllab and rltools repo manually and added them to the PYTHONPATH as mentioned. I also installed all depenedencies from environment.yml in rllab.

However, when I run the command:
python3 runners/run_multiwalker.py rllab --control decentralized --policy_hidden 100,50,25 --n_iter 200 --n_walkers 2 --batch_size 24000 --curriculum lessons/multiwalker/env.yaml

I get the following errors:
Traceback (most recent call last):
File "runners/run_multiwalker.py", line 10, in
from madrl_environments.walker.multi_walker import MultiWalkerEnv
File "/Users/anirudhajitani/Downloads/Projects/MADRL/madrl_environments/init.py", line 1, in
from rltools.util import EzPickle, stack_dict_list
File "/Users/anirudhajitani/Downloads/Projects/MADRL/rltools/rltools/util.py", line 157
proxy.client("set_state", state_str, async=True)
^
SyntaxError: invalid syntax

I tried to find the the function client, but the prototype for it is completely different. Can somebody tell me what I am foing wrong?

Tensorflow shape error

Hello,

I'm using your MADRL code, with tensorflow 1.0. I don't know if you have any idea how to fix this issue? At least I don't know which shape should be correct (the (48, 64) one, or the (?, 32) one). Thank you!

The command I'm running: python run_multiwalker.py rllab --control decentralized --policy_hidden 100,50,25 --n_iter 200 --n_walkers 2 --batch_size 24000 --curriculum lessons/multiwalker/env.yaml

Traceback (most recent call last):
File "/home/local/yzha3/pycharm-2016.3.2/helpers/pydev/pydevd.py", line 1596, in
globals = debugger.run(setup['file'], None, None, is_module)
File "/home/local/yzha3/pycharm-2016.3.2/helpers/pydev/pydevd.py", line 974, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/home/local/yzha3/research/planning/MADRL/runners/run_multiwalker.py", line 52, in
main(RunnerParser(ENV_OPTIONS))
File "/home/local/yzha3/research/planning/MADRL/runners/run_multiwalker.py", line 48, in main
run()
File "/home/local/yzha3/research/planning/MADRL/runners/rurllab.py", line 271, in call
self.algo.train()
File "/home/local/yzha3/research/planning/MADRL/rllab/sandbox/rocky/tf/algos/batch_polopt.py", line 112, in train
paths = self.obtain_samples(itr)
File "/home/local/yzha3/research/planning/MADRL/rllab/sandbox/rocky/tf/algos/batch_polopt.py", line 98, in obtain_samples
return self.sampler.obtain_samples(itr)
File "/home/local/yzha3/research/planning/MADRL/rllab/sandbox/rocky/tf/samplers/vectorized_sampler.py", line 57, in obtain_samples
actions, agent_infos = policy.get_actions(obses)
File "/home/local/yzha3/research/planning/MADRL/rllab/sandbox/rocky/tf/policies/gaussian_mlp_policy.py", line 170, in get_actions
means, log_stds = self._f_dist(flat_obs)
File "/home/local/yzha3/research/planning/MADRL/rllab/sandbox/rocky/tf/misc/tensor_utils.py", line 8, in run
return sess.run(outputs, feed_dict=dict(list(zip(inputs, input_vals))))
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 767, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 944, in _run
% (np_val.shape, subfeed_t.name, str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (48, 64) for Tensor u'policy/mean_network/input/input:0', which has shape '(?, 32)'

DQN

I noticed that your paper has graphs produced from average DQN rewards, but it seems to me that something is not working correctly with your DQN algorithm. Also, when I try to apply the algorithm to the pursuit environment, there is a problem with the rank of your tensors being to low. I also see #TODO for the pursuit action spaces. Are there any missing commits or any directions you can point me in to get DQN from rltools running on the pursuit environment?

rltools serial.py error

File "runners/run_pursuit.py", line 73, in main
    run()
  File "/home/varun/projects/MADRL/runners/rurltools.py", line 221, in __call__
    blend_eval_trajs=self.args.blend_eval_trajs)
  File "/home/varun/projects/MADRL/rltools/rltools/algos/policyopt.py", line 38, in train
    iter_info = self.step(sess, itr)
  File "/home/varun/projects/MADRL/rltools/rltools/algos/policyopt.py", line 51, in step
    trajbatch0, _ = self.sampler.sample(sess, itr)
  File "/home/varun/projects/MADRL/rltools/rltools/samplers/serial.py", line 72, in sample
    timesteps_sofar += np.sum(map(len, ag_trajs))
TypeError: unsupported operand type(s) for +=: 'int' and 'map'

When I am executing the code for Pursuit evasion domain using rltools, I get the above error.

It belongs to this block of code:

timesteps_sofar = 0
        while True:

            ag_trajs = decrollout(self.algo.env, self.algo.policy, self.max_traj_len,
                                  self.algo.policy.action_space)
            print('AG_trajs: {}'.format((ag_trajs)))
            trajs.extend(ag_trajs)
            print('traj_map: {}'.format(map(len, ag_trajs)))
            timesteps_sofar += np.sum(map(len, ag_trajs))
            # print('timesteps_sofar: {}'.format(timesteps_sofar))
            if timesteps_sofar >= self.n_timesteps:
                break

I am running the code using this command: python runners/run_pursuit.py rltools --n_evaders 1 --n_pursuers 1

model monitoring

thank you for your codes
I can't install monitoring model which is in madrl environment.
the error:
pip install gym monitoring
Requirement already satisfied: gym in ./anaconda3/lib/python3.6/site-packages (0.10.9)
Collecting monitoring
Using cached https://files.pythonhosted.org/packages/a8/db/e8118fe34d232da4834ef49fe9a191a933911968e6e795936fc490920e09/monitoring-0.9.10.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "/private/var/folders/w2/yh_z8t9d09qbs732pclkqh8m0000gn/T/pip-install-ll5v0fgh/monitoring/setup.py", line 5, in
VERSION = eval(filter(lambda :.startswith("version"), file("Sources/monitoring/init.py").readlines())[0].split("=")[1])
NameError: name 'file' is not defined

Some problems about 'config.py'

I ran this code and countered this problem.
Creating your personal config from template... Personal config created, but you should probably edit it before further experiments are run Exiting.

I have checked config.py, but I do not know how to fix this.

Can you tell me how to use DDPG run multi-walker?

modul not found

hi
i have following error message when run run_multiwalker.py code.
"Traceback (most recent call last):
File "runners/run_multiwalker.py", line 7, in
from runners import RunnerParser
ModuleNotFoundError: No module named 'runners' "

Box2D.b2 error

I run my code in pycharm and in linux:
File "/home/niloofar/Desktop/MADRL-master/madrl_environments/walker/multi_walker.py", line 11, in
from Box2D.b2 import (circleShape, contactListener, edgeShape, fixtureDef, polygonShape,
ImportError: No module named 'Box2D.b2'

what should I do?

Get Action Policy Pursuit

I'm trying to display the agent's actions using these lines:

act_fns = [lambda o: policy.get_action(o)[0]] * len(self.env.agents)
print(act_fcs)

but i get that " [<function RLLabRunner.call.. at 0x7fb68ce15400>, <function RLLabRunner.call.. at 0x7fb68ce15400>]"
when executing !!
Do i miss something ?

MIT License

Hey, I'm working with the environments from this library, and have forked them to make some changes and would like to rerelease them as part of the supplemental materials of my paper. For that to be fully legal, this code would have to be released under an open source license (currently it's under no license). An MIT license is the simplest and easiest to work with in terms of others reusing your code. Could you please add that to this library real quick to make the university IP lawyer happy?