sebastianscherer88/deep_learning_library

A collection of DIY implementations of some common neural network architectures

deep_learning_library's Issues

create testing script for network class

Copy the testing cell from the original jupyter notebook to create a script that test the network class on the mnist data set. This script should be updated with every modification made to the network classes and used to test before any commits on the dev branches are made.

think about handling regression as well as classification

this could be handled by:

• passing a 'type' flag an init stage
• ensuring that final layers activation and cost function 'pair up' (i.e. resolve to give DL/DZ_{final_layer} = P- Y )
• introducing squared cost function
• adding regression mode for prediction method

add batches/epoch counter while training

while printint status messages during network training, include the number of batches per epoch to give a better indication of training progress within current epoch, i.e.

epoch: 2
batch: 13/1000

instead of (currently)
epoch: 2
batch: 13

implement different weight optimization technique

The idea is to allow for more than the generic gradient descent algorithm when it comes to network weight optimization. When building a network the user should be allowed to specify which optimization algorithm he/she wants to use to optimize the network's weights. They should have the option to toggle L2 regularization.

Possbile options:

• Generic gradient descent
• Momentum
• Nesterov accelerated gradient
• Adagrad
• Adadelta
• RMSprop
• Adam
• AdaMax
• Nadam
• AMSGrad

This could be done by introducing respective optimization object classes and attach one to each network's layer. That way it has direct access to the layer's weight's gradient and can manage the weight update on a low level, with the network class only supervising the weight update step from a high level.