NEORL (NeuroEvolution Optimisation with Reinforcement Learning) is a set of implementations of hybrid algorithms combining neural networks and evolutionary computation based on a wide range of machine learning and evolutionary intelligence architectures. NEORL aims to solve large-scale optimisation problems relevant to operation & optimisation research, engineering, business, and other disciplines.
NEORL can be used for multidisciplinary applications for research, industrial, academic, and/or teaching purposes. NEORL can be used as a standalone platform or an additional benchmarking tool to supplement or validate other optimisation packages. Our objective when we built NEORL is to give the user a simple and easy-to-use framework with an access to a wide range of covering both standalone and hybrid algorithms in evolutionary, swarm, supervised learning, deep learning, and reinforcement learning. We hope our implementation will allow beginners to enjoy more advanced optimisation and algorithms, without being involved in too many theoretical/implementation details.
Documentation is available online: https://neorl.readthedocs.io/en/latest/index.html
This repository and its content are copyright of Exelon Corporation © in collaboration with MIT Nuclear Science and Engineering 2021. All rights reserved.
You can read the first successful application of NEORL for nuclear fuel optimisation in this News Article.
Features | NEORL |
---|---|
Reinforcement Learning (standalone) | ✔️ |
Evolutionary Computation (standalone) | ✔️ |
Hybrid Neuroevolution | ✔️ |
Supervised Learning | ❌ |
Parallel processing | ✔️ |
Combinatorial/Discrete Optimisation | ✔️ |
Continuous Optimisation | ✔️ |
Mixed Discrete/Continuous Optimisation | ✔️ |
Ipython / Notebook friendly | ✔️ |
Detailed Documentation | ✔️ |
Advanced logging | ✔️ |
Optimisation Benchmarks | ✔️ |
Note: despite the simplicity of NEORL usage, most algorithms, especially the neuro-based, need some basic knowledge about the optimisation research and neural networks in supervised and reinforcement learning. Using NEORL without sufficient knowledge may lead to undesirable results due to the poor selection of algorithm hyperparameters. You should not utilize this package without some knowledge.
Note: NEORL supports Tensorflow versions from 1.8.0 to 1.14.0 (Recommended). Please, make sure to have the proper TensorFlow installed on your machine before installing NEORL.
NEORL requires python3 (>=3.5) with the development headers. You'll also need system packages CMake, OpenMPI and zlib. Those can be installed as follows
sudo apt-get update && sudo apt-get install cmake python3-dev
To install NEORL on Windows, it is recommended to install Anaconda3 on the machine first to have some pre-installed packages, then open "Anaconda Prompt" as an administrator and use the instructions below for "Install using pip".
For both Ubuntu and Windows, you can install NEORL via pip
pip install neorl
Here is a quick example of how to use NEORL to minimize a 5-D sphere function:
#---------------------------------
# Import packages
#---------------------------------
import numpy as np
import matplotlib.pyplot as plt
from neorl import DE, XNES
#---------------------------------
# Fitness
#---------------------------------
#Define the fitness function
def FIT(individual):
"""Sphere test objective function.
F(x) = sum_{i=1}^d xi^2
d=1,2,3,...
Range: [-100,100]
Minima: 0
"""
return sum(x**2 for x in individual)
#---------------------------------
# Parameter Space
#---------------------------------
#Setup the parameter space (d=5)
nx=5
BOUNDS={}
for i in range(1,nx+1):
BOUNDS['x'+str(i)]=['float', -100, 100]
#---------------------------------
# DE
#---------------------------------
de=DE(mode='min', bounds=BOUNDS, fit=FIT, npop=50, CR=0.5, F=0.7, ncores=1, seed=1)
x_best, y_best, de_hist=de.evolute(ngen=120, verbose=0)
print('---DE Results---', )
print('x:', x_best)
print('y:', y_best)
#---------------------------------
# NES
#---------------------------------
x0=[-50]*len(BOUNDS)
amat = np.eye(nx)
xnes=XNES(mode='min', bounds=BOUNDS, fit=FIT, npop=50, eta_mu=0.9,
eta_sigma=0.5, adapt_sampling=True, seed=1)
x_best, y_best, nes_hist=xnes.evolute(120, x0=x0, verbose=0)
print('---XNES Results---', )
print('x:', x_best)
print('y:', y_best)
#---------------------------------
# Plot
#---------------------------------
#Plot fitness for both methods
plt.figure()
plt.plot(np.array(de_hist), label='DE')
plt.plot(np.array(nes_hist['fitness']), label='NES')
plt.xlabel('Generation')
plt.ylabel('Fitness')
plt.legend()
plt.show()
NEORL offers a wide range of algorithms, where some algorithms could be used with a specific parameter space (e.g. discrete only).
Algorithm | Discrete Space | Continous Space | Mixed Space | Multiprocessing |
---|---|---|---|---|
PPO | ✔️ | ✔️ | ❌ | ✔️ |
A2C | ✔️ | ✔️ | ❌ | ✔️ |
DQN | ✔️ | ❌ | ❌ | ❌ |
ES | ✔️ | ✔️ | ✔️ | ✔️ |
DE | ❌ | ✔️ | ❌ | ✔️ |
PSO | ✔️ | ✔️ | ✔️ | ✔️ |
XNES | ❌ | ✔️ | ❌ | ✔️ |
GWO | ❌ | ✔️ | ❌ | ✔️ |
PESA | ✔️ | ✔️ | ✔️ | ✔️ |
PESA2 | ❌ | ✔️ | ❌ | ✔️ |
All unit tests in NEORL can be run using pytest runner. If pytest is not installed, please use
pip install pytest pytest-cov
Then run
neorl --test
1- Radaideh, M. I., Wolverton, I., Joseph, J., Tusar, J. J., Otgonbaatar, U., Roy, N., Forget, B., Shirvan, K. (2021). Physics-informed reinforcement learning optimization of nuclear assembly design. Nuclear Engineering and Design, 372, p. 110966.
2- Radaideh, M. I., Shirvan, K. (2021). Rule-based reinforcement learning methodology to inform evolutionary algorithms for constrained optimization of engineering applications. Knowledge-Based Systems, 217, p. 106836.
3- Radaideh, M.I., Forget, B., Shirvan, K. (2021). Application of Reinforcement Learning Optimisation Methodology to BWR Assemblies ," In: Proc. International Conference on Mathematics & Computational Methods Applied to Nuclear Science & Engineering (M&C 2021), Rayleigh, North Carolina, USA, October 3-7, 2021.
To cite this repository in publications:
@misc{neorl,
author = {Radaideh, Majdi I. and Seurin, Paul and Wang, Haijia and Shirvan, Koroush},
title = {NEORL},
year = {2021},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/mradaideh/neorl}},
}
NEORL is currently maintained by Majdi Radaideh (aka @mradaideh), Paul Seurin (aka @pseur), Koroush Shirvan (aka @kshirvan)
Important Note: We do not do technical support and we do not answer personal questions per email.
To any interested in making NEORL better, there is an open undergraduate research position at MIT, see Be Part of the Machine Learning-based Neuroevolution Optimization Framework
NEORL was established in MIT back to 2020 with feedback, validation, and usage of different colleagues: Issac Wolverton (MIT Quest for Intelligence), Joshua Joseph (MIT Quest for Intelligence), Benoit Forget (MIT Nuclear Science and Engineering), Ugi Otgonbaatar (Exelon Corporation).