wouterkistemaker/neural-network

A multi-layer, feedforward neural network (FNN) implementation in Java

Neural-Network

A multi-layer, feedforward neural network (FNN) implementation in Java

In this branch, I decided to completely recode the project as I had lost track of where I left it behind since I stopped working on it for a few months. I decided to include functionalities that allow one to visualize the architecture of the Neural Network one has created.

An example of how this looks is shown below:

neural-network's Issues

Implement Xavier initialization

To allow maximum flexibility in initialization the weights, the Xavier initialization is recommended. Therefore, this algorithm has to be implemented.

Change backpropagation to allow multiple activation- and errorfunctions

Right now, the back-propagation algorithm (and mostly the updating of the weights) is made such, that using other activation-functions than the Sigmoid Transfer Function isn't really working. This is also the reason why all other activation-functions are currently annotated @Deprecated

This means the following method will need revision in the future, to allow more flexibility in usage:

private void updateWeights(Layer layer, Layer previous) {
        for (Neuron n : previous.getNeurons()) {

            for (Neuron next : layer.getNeurons()) {
                final NeuronConnection connection = n.getConnectionWith(next);

                final double delta = next.getDelta();
                final double previousOutput = n.getValue();

                final double deltaWeight = -learningRate * delta * previousOutput;

                connection.adjustWeight(deltaWeight);
            }
        }
    }

Change implementation of the Bias Neurons

Currently the way that the bias is taken into account (in the weighted sum) isn't accurate. I am still figuring out the key role of the Bias Neuron. Although I have added them in the code already, making use of it is discouraged.

BinaryCrossEntropyCost converges to NaN

The BinaryCrossEntroyCost-function seems to be outputting NaN in both the normal function as also its derivative. Further investigation is needed to understand this behaviour.

Vanishing gradient problem

The mathematics behind the Neural Network allow the vanish gradient problem to take place. A fix would be to have other activation- and errorfunctions (?)

To allow that, #2 will need to be fixed as well.

See #6, different initializations will also reduce the risk of this problem.

Change the initialization of the network

While fixed initialization of the weights is currently available, this from now on is scheduled for removal, as this doesn't affect the outcome of the network in a constructive way. In fact, it can only lead to a symmetric network. [1]

Next to this part being removed, a new way of initialization should be included that ensures that all initial weights are within set boundries, to minimize the risk for #5 and the 'exploding gradient problem'. Another option would be 'Xavier initialization' [1]

[1] Neural Network: Breaking The Symmetry | by Luthfi Ramadhan | Dec, 2020 | Towards Data Science n.d. https://towardsdatascience.com/neural-network-breaking-the-symmetry-e04f963395dd (accessed December 3, 2020).

Add documentation to emphasize that the activation-function is crucially related to outcome and input

As described in the title, the activation-function is intrinsically important for the network's input and outcome.

Looking at the sigmoid function, which is described as:

public Double apply(Double x) {
        return 1 / (1 + (Math.exp(-x)));
}

one might realize that after computing the limits of s(x) as x goes to ∞ and as x goes to -∞, this function will always result in a value between 0 and 1. This means that using this function as activation-function, a network can never be trained to output 3, or 7, or -9.3 as these are all outside the bounds of 0 ≤ x ≤ 1.

Make sure to add this in the wiki/documentation!