saltierl/saltie

🚗 Rocket League Distributed Deep Reinforcement Learning Bot

Saltie

Saltie is a deep reinforcement learning bot and framework. It learns how to play Rocket League by receiving rewards for certain actions.

Its backend framework for communicating with the game is RLBot. RLBot is a framework to create bots to play Rocket League that reads values from the game and outputs button presses. RLBot works for up to 10 bots.

Requirements

Windows (7+), Rocket League, Python 3, Tensorflow.

Short Description

Saltie is a bot that uses Neural Networks and Machine Learning to learn how to play the game. It also has tools for training bots and collecting the replays from a lot of distributed computers

Setup

For setup instructions, check out the wiki.

Contributing

We are currently looking for contributors! If you would like to contribute, join the Discord server and let one of the Team Leads know.

You can also start a pull request and we will review the changes.

DEMO

saltie's Issues

RawConfigParser in master having errors

E:\RLBOT\Saltie-tournament2/bot_code/training does not exist creating
gameName: 1519417328989-815in E:\RLBOT\Saltie-tournament2/bot_code/training/replays
Traceback (most recent call last):
File "E:\RLBOT\Saltie-tournament2\runner.py", line 243, in
main()
File "E:\RLBOT\Saltie-tournament2\runner.py", line 180, in main
bot_parameter_list.append(bot_config[BOT_CONFIG_AGENT_HEADER])
AttributeError: RawConfigParser instance has no attribute 'getitem'

Change critic to have an output of -1 to 1

Filter Downloaded replays

The ability to download and filter replays based on criteria
(model hash, version info, upload date)

New feature of timers

Add 2 timers
time since bOnGround turns to false (counts up to 1.45 then resets back to 0 once you land on ground)
time since bDoubleJumped (counts up to 1.0 then resets back to 0 once bDoubleJumped is turned false)

Allow variables to be saved with more fidelity.

Allow input layers, hidden layers and output layers to be stored in different tensorflow savers.

And different model groups to also be saved in different saves (actor vs critic)

Speed up downloading

zipping in groups of a 100 files then downloading that group then unzipping on the client

another option could we open up a socket for downloading then having the client piece together the files. (this is probably the fastest but most complicated to create)

Make sure the copy trainer works for keras

Add a range of random packets needed to force certain tutor bot controls

Basically we put the bot in situtaions where it would jump or it would use handbrake or it would use boost or halfflip or whatever

Add a message to tell user that upload will work

Maybe talk to the server in runner.py to know that you can talk to the server.

We could also use that point to download any new models and config files

Reward_Trainer missing code

in reward_tranier.py i was getting the error, below. sorry, could not figure out how to do a pull request?

C:\Users\Lokiwizz\AppData\Local\Programs\Python\Python36\python.exe C:/Users/Lokiwizz/Saltie/bot_code/trainer/reward_trainer.py Traceback (most recent call last): File "C:\Users\Lokiwizz\AppData\Local\Programs\Python\Python36\lib\configparser.py", line 1138, in _unify_values sectiondict = self._sections[section] KeyError: 'Model Configuration

adding this to the file resolved that problem.

def get_config_name(self):
return 'reward_trainer.cfg'

Add mini batching

Basically we write to the gpu in batches of (some large number like 4k)
then inside gpu we run training on batches of 100 (or something)

I think because it is reinforcement learning training should be in batches based on seconds of in game time. Not actual sample numbers.

Create sequence of random controls

This will be things like
Go to position X,Y,Z on the field.
Or randomly slide for 5 seconds or randomly jump every 2 seconds
Basically this should be a sequence of controls that will randomly occur to add more variety but also stability than a single frame of random controls.

Provide automatic evaluation of the newest model

Change model hashing method to file hash

Make the client keep the file hash around until replay upload, and send the first 8 chars with the request.

On model upload, the server will hash the file before making available to clients. When a replay comes in, it will match the client model with the models on disk.

Fix the sending of variables for displaying debug info

Variables for the model debugger need to be fixed to send in the correct order.

Move the Keras Rocket League Replay Reader to saltie

Allow client to match replay file with model hash

Convert to tensorflow multinominal instead of numpy multinominal

Create clear documentation for anyone to run saltie

Create policy model

Given the position of 1,3,5 players + ball have the model try and predict the position of the other player.

Get the last set of controls and normalize error based on control rates

This will basically get how much of each control was in the last batch and should normalize loss by the control.
If it is a rare control it should get a huge loss if wrong. A common control should get a much smaller loss

Download trainer use a different port/server

Create config file for changing model.

Currently inside saltie.py the model has to be hard coded in.

It would be better if we could do this inside the config file for the bot.

add timing to the replays stored

but maybe outside of the current data stored?

Write model params to file to save with model

This makes it easier to remember how to run it

Reading Inventory

Hello,

I want to get all my items which I own with the amount/ paint/ cert in one file. Is it possible to read it with the Injector or is that impossible?

Cheers!

Create a list of savers

So instead of saving and restoring all of the critic network or actor network we should make it so there is a lot more customization for saving networks so more models can be used

Add normalization to all inputs

Basically scales all inputs to be between 0 and 1 with a middle point of 0.5

Scaling should be done in a linear fashion

Improve Process of downloading a model

Right now it should work but it needs to be tested and double checked on how it will work.
(Maybe see if we can make this work for the tournament too?)

And make sure that it works with the config options.

---

Add ability of filtering/selecting one model to download from multiple possible models

Create a config file for the trainer

This could be used to change hyper parameters and to choose what model and what kind of training will be performed.

Currently there are 3 kinds of training thought of
A copy trainer that always tries to copy inputs. Not model specific

A reinforcement trainer that applies rewards over time. This is model specific. The model that generated the data has to also be the one training on it.

A copy reinforcement trainer. Not model specific. This copies inputs but only for the ones that the bot should be rewarded for doing.

Write more information about game to the file version information

In the file versioning we should write the number of participants and how many players team 0 has.

Client side way to know if the server is responding correctly

A heartbeat or something from the client to know if it will upload successfully

Make a gui to choose training and config options

This would make it easier to save config sets so it is easier to run trainers in the future

Correct hashing of model

Model hashing is currently broken and does not correctly hash the variables.

Create a config for evaluation mode

This can just be a boolean on if the bot is in evaluation mode or not

Live graph needs a runner at the top level

E:\RLBOT\Saltie-tournament2\bot_code\livedata>live_graph.py
Traceback (most recent call last):
  File "E:\RLBOT\Saltie-tournament2\bot_code\livedata\live_graph.py", line 1, in <module>
    from bot_code.livedata import live_data_util
ImportError: No module named bot_code.livedata

Correctly split up reg loss

This should be more specific to each part of the network for actor

Be able to read version 4 replays

Version 4 replays are no longer readable bc they have less features.
We need to add a way to convert the old array size to the new array size.

Send down models/configs based on user

For different user info (ips, upload names)
Send down different configs and different models.

add logger

https://docs.python.org/3.6/library/logging.html

This will make it easier to set a debug mode and only log debug animations

Add a way to upload a model to the server after it has been trained.

upload files in separate thread, delete after upload

We currently write file data and upload in the same thread as the one the bot logic is running in.

I think we can write the file data in the same thread but uploading the file data needs to happen in a separate thread.

---

Files that upload successfully should be deleted from the local disk.

Change to using tensorflow loss instead of custom

https://www.tensorflow.org/api_docs/python/tf/nn/l2_loss

Prepare the bot for the tournament

We need to make sure that saltie still works without the custom bot_manager.py

And create a way for saltie to download the correct files that are needed.
(Maybe some sort of setup.py that downloads the correct files?)

An ability to download certain configs to certain users

Either explicitly to some users or maybe have some sort of group people can opt into.

(I think having a group might be better?)
example
have user opt into relief bot group where they get that bot
or they can opt into a genetic algorithm group

delete uploaded files

Files that upload successfully should be deleted from the local disk.

Batch download of files for replays for training

Fix accuracy testing for regression

regression does not currently match what is actually happening in the bucketing.