mauicv / gerel Goto Github PK
View Code? Open in Web Editor NEWevolutionary algorithms for reinforcement learning
License: MIT License
evolutionary algorithms for reinforcement learning
License: MIT License
Add some functionality to pretrain network to fit certain inital conditions. This is in order to solve the problem where all the networks within a certain population end up exhibiting similar behavours.
Add function that creates a genomes_seeder or population from DataStore and specified generation.
If series runs of batches of environments is more efficent then we don't need to store the entirity of the population data in memory. This is dependent on what we're using to simulate the environment.
Mutator Class should a) be an interface
from src import Population
from src import Mutator
from src import generate_neat_metric
from src import Model
class CustomlMutator(Mutator):
def mutate(self, genome):
# do something to genome
def crossover(self, genome):
# crossover genome
custom_mutator = CustomMutator()
.
.
.
custom_mutator(population)
Questions:
The aspect of evolutionary algorithms applied to RL that benifits from multi processing is really the environment simulation step that has to be performed for each genome in the class. This is outside the scope of what this library tries to do but...
Currently datastore saves the genome structure as all nodes and edges includeing input and output nodes but from_genes hydrator factory expects num_inputs and num_outputs and that the nodes and edge lists passed are only hidden nodes. Hence to use from_nodes with datastore we have to do:
generation = ds.load(generation_in)
nodes, edges = last_gen['best_genome']
input_num = len([n for n in nodes if n[4] == 'input'])
output_num = len([n for n in nodes if n[4] == 'output'])
nodes = [n for n in nodes if n[4] == 'hidden']
genome = from_genes(
nodes, edges,
input_size=input_num,
output_size=output_num,
weight_low=-2,
weight_high=2,
depth=len(LAYER_DIMS))
Similarly computing the number of layers is not trivial and should be just be a value stored in some meta data either on the generation or the DataStore class itself.
Currently we're using a dict to store the data on each species this should be abstracted into a class becuase at the very least it'll enable code completion.
from_genes
should probabaly be renamed to genome_from_genes
added to the Genome class as static method. Same is true of other factory methods.
Mutator Class should act on populations. This would look like:
from src import Population
from src.Neat import NEATMutator
from src import generate_neat_metric
from src import Model
def compute_fitness(genome):
model = Model(genome)
# compute fitness of model here
return fitness
mutator = NEATMutator()
metric = generate_neat_metric()
population = Population(metric=metric)
for i in range(10):
for genome in population.genomes:
genome.fitness = compute_fitness(genome.to_reduced_repr)
population.speciate()
mutator(population)
Genome.default()
builds a very simple Genome with one hidden node. The assumption is that we use it in NEAT algorithm.
Should really be called Genome.default_neat()
and we should at some point implement other factories...
Currently pyg
will successfully run NEAT even with some minor errors #5. The aim is to build pyg so that:
Same as REINFORCE-ES #19 but we select the highest perfoming solution from population that lies along the derivative vector rather than taking a fixed step size each time.
Seems like running populations around ~300 with network hidden layer sizes ~ [100, 100] ends up resulting in significant slow downs. I don't feel these sizes should be an issue. Figure out what cuases this? Is it just memory?
Population Class should be an interface with the Speciate method to be implemented.
This looks like:
class CustomPopulation(Population):
def speciate(self):
for genome in self.genomes:
# do something here
p = CustomPopulation(metric=NEAT_metric())
Simple mutator just selects the top n genomes and mutates each some number of times to refil the population.
The NEAT mutator is supposed to kill off species if they havn't imporved in some number of generations.
This may be an error that occurs. Very intermittent so mostly leaving here in case it crops up so as to provide more notes on. Do not persue as may not exist!
How do we consolodate the NEAT method of selecting weights and the REINFORCE-ES method?
Instead of having integration tests replace with Example folder and include documentation for each case.
Looks something like:
p = Population(weight_init_fn=Uniform(-2,2),
bias_init_fn=Uniform(0, 1))
related to #13
see this blog for available options
Add meta data to datastore implementation.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.