vhranger/rnnlib

A library for recurrent neural networks in Dlang

rnnlib

A library for recurrent neural networks in Dlang

The aim of this project is to build a RNN factory such that one can construct RNN architecture using simple layers and any non linear function.

The neural networks will primarily be trained using Evolutionnary Algorithms. The reason why we do not use any optimization based on gradient descent is to avoid many issues it gives us when optimizing a rnn like vanishing/exploding gradients (e.g. see On the difficulty of training recurrent neural networks).

One of the auxiliary goal of this project is to cut any dependencies between the evolutionary algorithms and the recurrent neural netwroks to enable anyone to use it independently.

Daily Todo

• Think about parameters shared between layers: can the layer be different and share the same parameters ?
• (optm) randclone weigths: serialized_data holds all weigths -> can be constructed using smaller array with random cloning.

TODO

• RNN
  • Matrices
    • Optimization
      • Use a single "dot" product and optimize it.
      • Matrix abstract multiplication: mult(auto override this T)
  • Vectors
    • Optimization
      • alias array for vector ?
      • use arrayfire ?
  • Layers
    • Linear
    • Functional
    • Recurrent
  • Neural Network *
• TRAINING
  • Evolutionary Algorithms
    • Evolution Strategy
    • Genetic Algorithm
    • Particle Swarm optimization
    • Ant colony optimization
  • Other Gradient-free Optimization
    • Nelder–Mead Simplex
    • DIRECT
    • DONE
    • Pattern Search
    • MCS
• TESTS
  • Functional Tests
    • Some good examples
  • Visualizable Tests
    • test the optimization algorithm for drawing graphs: force directed layout gives a function.
  • Machine Learning Tests
    • Adding Problem
    • XOR
    • Copying memory
    • Pixel-by-pixel MNIST (+ permuted)
    • NLP ?
    • Music Generation ?
  • Deep Reinforcement Learning Final Tests
    • Gym
    • Universe
    • Zbaghul
• DOCUMENTATION
  • Vector
  • Matrix
  • Layer
  • Neural Network
  • Optimization Algorithms
    • EA
    • OGFO
• FORMAT
  • CodeCov
  • Travis CI ?
  • Command Line git (+ .gitignore)
  • dub
  • For most of these