cool-rr/marley

A framework for multi-agent reinforcement learning.

Marley

Marley is a framework for multi-agent reinforcement learning. It lets you design a game and run experiments with multiple AI-powered agents playing that game.

Marley isn't ready for use yet; it works for my research purposes, but I haven't yet made the work needed to make it easy to understand for others. If you're still interested in using it, feel free to read the code or email me for help.

I'm using Marley in my research mission to use machine learning to understand human society. I want to produce emergent reciprocation in selfish RL agents and use that to show social behavior. Talk video, slides, sign up for updates.

I hope to make Marley easier to use soon.

GridRoyale

View the live version here!

GridRoyale is a life simulation. It's a case study that's bundled with Marley.

GridRoyale is similar to Game of Life or GridWorld, except I added game mechanics to encourage the players to behave socially. These game mechanics are similar to those in the battle royale genre of computer games, which is why it's called GridRoyale.

Players get 10 points for eating food, i.e. the dots that regenerate randomly. Players lose 5 points for colliding with other agents or walls. Players lose 10 points for being shot.

How to run GridRoyale

Installation:

$ pip install marley

Run the server:

$ marley grid_royale demo

This will automatically open a browser window and show you your simulation.

marley's Issues

Research temporal-difference learning

Research off-policy learning

Make the algorithm actually produce smart behavior

Document game rules in English

Requesting as per sprint discussion - these rules currently aren't written up?

Fix bug with player walking into oncoming bullets

There's a bug I observed: When there are a bunch of bullets one after another, and the player walks towards them, sometimes some of the bullets don't hit the player, but skip past it. Figure it out and fix it.

Core logic

Hello there

Im very intrigued by the idea of this project. I was thinking that maybe the core logic of it is a bit too "biological" for being basis of "social" dynamics.

Don't get me wrong in being sentimental, let me explain what i mean. Of course the struggle to avoid "death" is encoded in the nature of any biological and living thing. And many social behaviors have emerged solely out if the struggle to survive or effectiveness of living in social groups.

But, joining a gang is a fatal decision that has been made by millions throughout history of mankind. The average age of death of gang members is 23 or something like that. The point is that decisions that individuals make might be recurring among individuals even if those decisions are guaranteed to kill you, one way or the other.

Bundle immutabledict instead of requiring it

Draw a square around the grid

Module-level docstrings

A brief docstring at the top of each module would make it easier for newbies to navigate the codebase.

Observation should be used to predict next observation instead of reward

Then get the reward from that. Maybe we need a few iterations of that.

AttributeError: module 'tensorflow' has no attribute 'python'

At master (526df2a), on Linux, Python 3.8.7, installed packages:

absl-py==0.11.0
astunparse==1.6.3
cachetools==4.2.0
certifi==2020.12.5
chardet==3.0.4
click==7.1.2
Flask==1.1.2
flatbuffers==1.12
gast==0.3.3
google-auth==1.24.0
google-auth-oauthlib==0.4.2
google-pasta==0.2.0
-e git+https://github.com/cool-RR/grid_royale@526df2a01fa8b0ba39423a6977929539c451673a#egg=grid_royale
grpcio==1.32.0
h5py==2.10.0
idna==2.10
immutabledict==1.2.0
itsdangerous==1.1.0
Jinja2==2.11.2
Keras==2.4.3
Keras-Preprocessing==1.1.2
Markdown==3.3.3
MarkupSafe==1.1.1
more-itertools==8.6.0
numpy==1.19.5
oauthlib==3.1.0
opt-einsum==3.3.0
protobuf==3.14.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
PyYAML==5.4.1
requests==2.24.0
requests-oauthlib==1.3.0
rsa==4.7
scipy==1.6.0
six==1.15.0
tensorboard==2.4.1
tensorboard-plugin-wit==1.8.0
tensorflow==2.4.1
tensorflow-estimator==2.4.0
termcolor==1.1.0
typing-extensions==3.7.4.3
urllib3==1.25.11
Werkzeug==1.0.1
wrapt==1.12.1

Starting server emits uncaught error:

(venv) zhuyifei1999@zhuyifei1999-ThinkPad-T480 /tmp/grid_royale $ grid_royale play
Traceback (most recent call last):
  File "/tmp/grid_royale/venv/bin/grid_royale", line 33, in <module>
    sys.exit(load_entry_point('grid-royale', 'console_scripts', 'grid_royale')())
  File "/tmp/grid_royale/venv/bin/grid_royale", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.8/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/grid_royale/grid_royale/__init__.py", line 9, in <module>
    from .base import *
  File "/tmp/grid_royale/grid_royale/base.py", line 74, in <module>
    from . import utils
  File "/tmp/grid_royale/grid_royale/utils/__init__.py", line 53, in <module>
    copyreg.pickle(tensorflow.python._tf_stack.StackSummary, pickle_stack_summary)
AttributeError: module 'tensorflow' has no attribute 'python'

Improve `measure` by including states in which the food isn't in a straight line

Remove `launch-config` from things Wing keeps on wpr

Add command for Blackjack demo

Allow disabling bullets

It would be cool if a command line flag could disable all shooting in the game, to make it just about eating food and avoiding collisions. There's some interesting behavior there.

Online documentation

Hey I made a interactive online documentation for your Grid_Royale project.
It is hosted at Website

Add an action to build a wall

Add an action to build a wall next to the player. A wall can't be passed by a creature, walking into it results in collision damage. It must be shot with a bullet.

This can be conducive to territory-building.

Optimization: Make all the creatures calculate their next step in parallel

Calling predict for each observation separately is a huge time-sink. We should batch these as much as possible. All creatures of the same strategy should batch their call to predict and then split the result up. Maybe we can also run multiple games in parallel for training.

Make all actions legal

It'll simplify the logic if all actions were considered legal. Right now the only actions that are not considered legal are walking toward a wall. We should make that a legal action, and just have that count as a collision with the usual negative reward.