An application to train reinforcement learning to control a chemostat containing multiple interacting populations of bacteria.
Has been tested on max OSX and ubuntu.
To use the package within python scropts, CBcurl
must be in PYTHONPATH.
To use the package from the command line CBcurl
must be in you $PATH variable
Standard python dependencies are required: numpy
, scipy
, matplotlib
.yaml
is required to parse parameter files. argparse
is required for the command line application. pytest
is required to run the unit tests. If you would like to use the neural network functionality then TensorFlow
is required. The lookuptable versions of the agents will work without 'TensorFlow' if the lookuptable_Q_learn function is imported into a script (see examples). Instructions for installing 'TensorFlow' can be found here:
https://www.tensorflow.org/install/
CBcurl
can be used in two ways:
-
Code files can be imported into scripts, see examples
-
By running from the command line:
$ CBcurl.py agent parameters [options]
positional arguments:
-
agent: L selects the lookuptable agent, N selects the neural agent
-
parameters: a .yaml file containing the parameters that define the system
optional arguments:
-
-s, --save_path: path to save results
-
-d, --debug: True or False, selects whether or not to print debug information
-
-r, --repeats: the number of time the training run should be repeated
Results will automatically be saved a directory structure will be created in the supplied save_path:
save_path
├── WORKING_data
│ ├── train
│ ├── train_survival.npy
│ ├── pops.npy
│ ├── train_rewards.npy
│ ├── time_sds.npy
│ └── reward_sds.npy
├── WORKING_graphs
│ ├── train
│ ├── train_survival.npy
│ ├── pops.npy
│ └── train_rewards.npy
├── WORKING_saved_network
├── Q_table.npy
├── state_action.npy
└── visited_states.npy
WORKING_data contains the numpy arrays of gathered training data. train contains population curve of a training episode at intervals given by the test_freq parameter, these are only saved if DEBUG = True. train_survival.npy contains moving averages of the survival times of the episodes in the training run, time_sds.npy contains the standard deviations of these moving averages. train_rewards.npy contains moving averages of the rewards of the episodes in the training run, reward_sds.npy contains the standard deviations of these moving averages. pop.npy contains the population of the final episode after the agent has been trained.
WORKING_graphs contains .png graphs of the data in WORKING_data.
WORKING_saved_network contains the saved Q network, only saved by a neural agent
Q_table.npy contains the final Q_table, only saved by a lookuptable_agent
state_action.npy conatins the state action array
visited_states.npy contains an array of the number of times each state was visited, only saved by a neural agent
The user can define a system and the training parameters by creating a .yaml parameter file. Examples of these for multiple two and three species systems are found in the examples/parameter_files directory.
The examples directory contains examples including the single auxotroph system, double auxotroph system and the smaller target system. These show how the functions can be imported into a script and used to carry out reinforcement learning on a system specified in a parameter file. These can be run without adding CBcurl to PYTHON_PATH. As they take a ling time to run, some of them have been run already and the results are in the results directory. These examples are run with python, for example:
$ python LT_single_aux_example.py
In the testing directory run
$ py.test
to run all unit tests.
The misc directory contains code that is not directly involved in the user facing application. This includes the code used for the simulations to find steady states, the analysis of the eigenvalues of the Jacobian at these steady states and plotting code.