danzimm/cerebrum

Following along at http://neuralnetworksanddeeplearning.com

cerebrum's Issues

Feed forward Neural Network

• Create FFNN with a variable amount of arbitrarily sized hidden layers
• Forward propagation to calculate new values
• Back propagation to enable training
• Initialize with preset weights

For now it will be assumed that every layer is completely connected to each other (that is every node in layer n is connected to every node in layer n+1 for every n).

Basic Matrix Library

• Muliplication
  • Matrix
    • In place
    • Copy
  • Scalar
    • In place
    • Copy lhs
    • Copy rhs
• Transpose(in place/create new)
  • In place
  • Copy
• Addition
  • Matrix
    • In place
    • Copy
  • Scalar (meaning form a matrix with all entries that element and then do normal matrix addition)
    • In place
    • Copy lhs
    • Copy rhs
• Subtraction
  • Matrix
    • In place
    • Copy
  • Scalar (meaning form a matrix with all entries that element and then do normal matrix subtraction)
    • In place
    • Copy lhs
    • Copy rhs
• Negation
  • In place
  • Copy
• Hadamard Product
  • In place
  • Copy

It doesn't make sense to have scalar hadamard product since that would be the same as scalar multiplication.

CNN

This is the issue tracking implementation of a CNN

Create cli to generate and train NN

The next step in visualizing these NN is to create a cli to generate and train NN

Optional is bloated

Optional currently has a whole sizeof(void*) in addition to the size of the value it stores. We should either:

• Change the void* to a bool so that we only append an extra byte
• Figure out how to have a tagged union so that we don't add any extra space at all

Run/test results should utilize emoji on supported platforms for the feels

Tests are great, tests make me feel great. And emoji make everyone feel great. I suggest that symbolic emoji such as ✅, ❗️, ❌, etc be used on platforms where emoji are supported.

Implement cross entropy cost function

C[x,w,b] = - 1 / n \sum_{x} [ y(x) ln(a(x, w, b)) + (1 - y(x)) ln (1 - a(x, w, b)) ]

Audit error messages

Currently in the logs I've seen some badly worded error messages. Need to fix that.

Add more activation functions (and tests for them)

• tanh
• Heaviside step function
• Gaussian
• ArcTan
• Sinc
• Sinusoid
• Soft Exponential

See this for more info

Properly Implement SoftMax

SoftMax regression should only really be used on the last layer of the network, and should be used with the log likelihood cost function. I want this to be statically compiled differently than if it wasn't a softmax neural network (so we can't pass a softmax flag at runtime). My current thinking is to have a custom cost function called softmaxlogliklihood. Then it just ignores whatever activation function is passed in and uses its own softmax function.

Web server mode on cli (Visualizations)

Finally we want to add a feature where you can create the NN using a web interface. When creating this web interface use websockets to communicate to the backend doing all the actual training. Use the features in #11 and #12 to enable a rich experience

RNN

This is the issue tracking the implementation of an RNN

NN Tests

• Train and test binary to decimal
• Train and test decimal to binary
• Train and test decimal recognition

Abstract out cost functions

• Cost function passed as meta-parameter to NN
• Cost function derivative passed as meta-parameter to NN

Weights and Biases Introspection API

In order to introduce visualizing the NN we need to be able to inspect and directly manipulate the weights and biases:

• Add const getter for weights
• Add const getter for biases
• Add setter for specific weight
• Add setter for specific bias

Callback API

We want to be able to visualize these neural networks. The first step of this is to add callbacks when certain events occur in the neural network:

• Minibatch finishes (pass which mini batch finished, out of how many mini batches)
• A new layer has been activated during training: pass which layer and the activations/z's
• A new layer has back propagated: pass which layer and the delta weights/biases