add convolutional neural net

[robertleeplummerjr]

add convolutional neural net

[robertleeplummerjr]

@riatzukiza I'd like to continue convolutional neural nets here. You showed interest, I'd love to work together to achieve this.

After researching this, I've come to the conclusion that the "convolutional" network isn't really a type of network, it is more an added intermediate layer between layers of an already defined type of network (I will insert here that I've been wrong before, and will be wrong again, and I could be now). That being said, I believe we can achieve this fairly easily by either embracing the ability to process them as either an array or a matrix (arguably just a larger flat array) with our existing codebase. What does this mean? Any network we have could be convolutional with minimal effort.

Speaking naively here...
At this point we have the following networks:

• Feedforward - new brain.NeuralNetwork()
• RNN - new brain.recurrent.RNN()
• LSTM - new brain.recurrent.LSTM()
• GRU - new brain.recurrent.GRU()
• More networks yay!

Imagine rather than having to do this (which increases our technological footprint exponentially):

• Feedforward - new brain.NeuralNetwork()
• RNN - new brain.recurrent.RNN()
• LSTM - new brain.recurrent.LSTM()
• GRU - new brain.recurrent.GRU()
• More networks... insert jaws theme here!
• Feedforward Convolutional - new brain.convolutional.NeuralNetwork()
• RNN Convolutional - new brain.convolutional.RNN()
• LSTM Convolutional - new brain.convolutional.LSTM()
• GRU Convolutional - new brain.convolutional.GRU()
• More networks... insert jaws theme here!

We simply do this:

• Feedforward - new brain.NeuralNetwork({ convolutional: true || false })
• RNN - new brain.recurrent.RNN({ convolutional: true || false })
• LSTM - new brain.recurrent.LSTM({ convolutional: true || false })
• GRU - new brain.recurrent.GRU({ convolutional: true || false })
• More networks yay!

Here we gain an optional on/off, and a clear distinction of what gives a neural network this capability.
Thoughts? More to come!

[robertleeplummerjr]

not yet. We'd love help, and are currently working on it.

[riatzukiza]

The way I've seen other packages do is to allow composition of networks by creating their layers, then threading those together. The above schema seems potentially restricting unless these implementations could provide huge performance gains over a compositional strategy.

…

On Fri, Jan 13, 2017 at 10:12 PM, Robert Plummer ***@***.***> wrote: @riatzukiza <https://github.com/riatzukiza> I'd like to continue convolutional neural nets here. You showed interest, I'd love to work together to achieve this. After researching this, I've come to the conclusion that the "convolutional" network isn't really a type of network, it is more an added intermediate layer between layers of an already defined type of network (I will insert here that I've been wrong before, and will be wrong again, and I could be now). That being said, I believe we can achieve this fairly easily be either embracing the ability to process them as either an array or a matrix (arguably just a larger flat array) with our existing codebase. What does this mean? Any network we have could be convolutional minimal effort. Speaking naively here... At this point we have the following networks: - Feedforward - new brain.NeuralNetwork() - RNN - new brain.recurrent.RNN() - LSTM - new brain.recurrent.LSTM() - GRU - new brain.recurrent.GRU() - More networks yay! Imagine rather than having to do this (which increases our technological footprint exponentially): - Feedforward - new brain.NeuralNetwork() - RNN - new brain.recurrent.RNN() - LSTM - new brain.recurrent.LSTM() - GRU - new brain.recurrent.GRU() - More networks... *insert jaws theme here*! - Feedforward Convolutional - new brain.convolutional.NeuralNetwork() - RNN Convolutional - new brain.convolutional.RNN() - LSTM Convolutional - new brain.convolutional.LSTM() - GRU Convolutional - new brain.convolutional.GRU() - More networks... *insert jaws theme here*! We simply do this: - Feedforward - new brain.NeuralNetwork({ convolutional: true || false }) - RNN - new brain.recurrent.RNN({ convolutional: true || false }) - LSTM - new brain.recurrent.LSTM({ convolutional: true || false }) - GRU - new brain.recurrent.GRU({ convolutional: true || false }) - More networks yay! Here we gain an optional on/off, an a clear distinction of what gives a neural network this capability. Thoughts? More to come! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#39 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKLqvSZX-VKwHciByvWcx0mx6KOpW8hhks5rSEtBgaJpZM4LE74V> .

[riatzukiza]

Could still compose entire networks, but I have not seen that with in the context of this package so far. The above strategy could work assuming that there were utilities for composing the networks. There was a streaming api, can these networks be piped into and out of one another?

…

On Sat, Jan 14, 2017 at 7:22 PM, Aaron Beavers ***@***.***> wrote: The way I've seen other packages do is to allow composition of networks by creating their layers, then threading those together. The above schema seems potentially restricting unless these implementations could provide huge performance gains over a compositional strategy. On Fri, Jan 13, 2017 at 10:12 PM, Robert Plummer ***@***.*** > wrote: > @riatzukiza <https://github.com/riatzukiza> I'd like to continue > convolutional neural nets here. You showed interest, I'd love to work > together to achieve this. > > After researching this, I've come to the conclusion that the > "convolutional" network isn't really a type of network, it is more an added > intermediate layer between layers of an already defined type of network (I > will insert here that I've been wrong before, and will be wrong again, and > I could be now). That being said, I believe we can achieve this fairly > easily be either embracing the ability to process them as either an array > or a matrix (arguably just a larger flat array) with our existing codebase. > What does this mean? Any network we have could be convolutional minimal > effort. > > Speaking naively here... > At this point we have the following networks: > > - Feedforward - new brain.NeuralNetwork() > - RNN - new brain.recurrent.RNN() > - LSTM - new brain.recurrent.LSTM() > - GRU - new brain.recurrent.GRU() > - More networks yay! > > Imagine rather than having to do this (which increases our technological > footprint exponentially): > > - Feedforward - new brain.NeuralNetwork() > - RNN - new brain.recurrent.RNN() > - LSTM - new brain.recurrent.LSTM() > - GRU - new brain.recurrent.GRU() > - More networks... *insert jaws theme here*! > - Feedforward Convolutional - new brain.convolutional.NeuralNetwork() > - RNN Convolutional - new brain.convolutional.RNN() > - LSTM Convolutional - new brain.convolutional.LSTM() > - GRU Convolutional - new brain.convolutional.GRU() > - More networks... *insert jaws theme here*! > > We simply do this: > > - Feedforward - new brain.NeuralNetwork({ convolutional: true || > false }) > - RNN - new brain.recurrent.RNN({ convolutional: true || false }) > - LSTM - new brain.recurrent.LSTM({ convolutional: true || false }) > - GRU - new brain.recurrent.GRU({ convolutional: true || false }) > - More networks yay! > > Here we gain an optional on/off, an a clear distinction of what gives a > neural network this capability. > Thoughts? More to come! > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#39 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AKLqvSZX-VKwHciByvWcx0mx6KOpW8hhks5rSEtBgaJpZM4LE74V> > . >

[robertleeplummerjr]

Good points!
In other libraries it was if they were saying: "Neural networks are so easy! All you have to do is know how they work, and then you can create one!"

When I first saw brain.js, I was shocked at the simplicity of:

var net = new brain.NeuralNetwork();
net.train();

Instantly I was enamored. Then dug further and found the toFunction method which runs the entire network in a single method. Some time later, I stumbled upon recurrent.js, and I thought: "Maybe we could do the same api in brain.js for a recurrent neural net!"

And after nearly a year, we have:

var net = new brain.recurrent.RNN();
net.train();

The reason that neural networks are hard to understand, I find, is because nearly every library abstracts the building of the neural network, when that can be done fairly easily once, and reused. The only thing the distinguishes each neural net from another (more or less) is what goes in the hidden layers.

In the case of the LSTM: https://github.com/harthur-org/brain.js/blob/master/src/recurrent/lstm.js
Composing a new neural net would simply be to know the math for your neural net, and extend RNN to use it. If one were interested in the architecture of the neural net, they could look at the internals of RNN and see them.

The idea that I think we are working towards is composition over inheritance. At this stage I would like to keep it as simple, and close to that principle as possible.

[robertleeplummerjr]

There was a streaming api, can these networks be piped into and out of one
another?

tl;dr: no
Long answer: There still is one. As it stands we are streaming data in, not delegating a hidden layer's output to another hidden layer or output via a stream.

[robertleeplummerjr]

@ddsol Is working on a formula composer that will be able to be reused between the nets. From what he has shown me, it will be consist of simple functional math, and before it runs, it compiles to a loop-less reusable function. He also is working on a gpu counterpart.

In the mean time, to get a simple poc convolutional neural net running with minimal effort running, I went ahead and did the following locally, though it is not yet fully working:

export default class CNN extends NeuralNetwork {
  /**
   *
   * @param input
   * @returns {*}
   */
  runInput(input) {
    this.outputs[0] = input;  // set output state of input layer

    let output = null;
    for (let layer = 1; layer <= this.outputLayer; layer++) {
      for (let node = 0; node < this.sizes[layer]; node++) {
        let weights = this.weights[layer][node];

        let sum = this.biases[layer][node];
        for (let k = 0; k < weights.length; k++) {
          sum += weights[k] * input[k];
        }
        this.outputs[layer][node] = 1 / (1 + Math.exp(-sum));
      }
      output = input = magicConvolutionFunction(this.outputs[layer]);
    }
    return output;
  }
}

The magicConvolutionFunction (not really its name) I'm still working on locally. I should have something pushed either tomorrow or Tuesday. I'm essentially reworking this https://github.com/karpathy/convnetjs/blob/master/src/convnet_layers_dotproducts.js#L43, understanding the logical flow, and simplifying to brain.js's standards. More will come soon!

[robertleeplummerjr]

I worked out the logical flow and simplified naming convention so almost anyone can make out what is going on in there.

You'll notice that it is essentially the same logic, the main difference I'm implementing a means of compiling a couple functions that will have all the math in it that has no loops, no instantiation of new objects/arrays, or no function calls within it. The upside to this is it is immensely faster.

The positive of it too is that the iteration can now be reused for the forward or runKernel can now be reused to build the backpropagate function or runBackpropagateKernel.

As for the main implementation of the Convolutional Neural Net. I've basically just extended the base NeuralNetwork and will attach it up to the Convolution when those methods/constructors materialize.

[robertleeplummerjr]

@riatzukiza, I believe I have seen the light of your statement. But I think we can make things both easily composable, easily configurable, and have a means of auto config. After a bit of tinkering, I can see how convolutions offer a much wider range of configurability than vanilla neural networks, and because of that we need to be able to define configurations. I've pondered on this for a while, but it may be pure trash if the community doesn't like it. I do plan on coming back around to the convolutional: true | false idea, but it may not be practical now that I've ripped into the idea more. Just playing around with syntax definitions, what do you think of this:?

new CNN({
  input: () => new Input(35, 35, 3),
  hiddenLayers: [
    (input) => new Convolution(input, {
      depth: 1,
      filters: 6,
      padding: 1,
      stride: 1
    }),
    Relu,
    (input) => new Pool(input, {
      padding: 1,
      stride: 1,
      width: 4,
      height: 4
    })
    40, //eventually backward compatible flat array, just for visuals
  ],
  output: (input) => new Output(tbd)
});

In this way we get:

• a flat list
• hierarchy of the structure of the hidden layers
• composability we like with js/es6
• a means of being able to create defaults for hiddenLayers when auto-configured using train() method if so desired
• potential backward compatibility with default layer types
• the ability to define with own type of layer

The reason for lambdas and usage of constructors: Constructors will be instantiated using new and will be passed the previous input. The more advanced lambda usage will receive the previous input, which can be passed into the layer being defined, along with options to define it, which is then returned to the lambda, which will be passed to the next constructor or lambda. Output is as yet to be determined.

thoughts?

[riatzukiza]

I like the functional arguements

[ddsol]

By passing the functions as arguments you can't figure out what's in it until you call it with actual data. This means you can't transcribe or optimize the entire function, only the parts.

I think it's better to be declarative and describe what the whole thing should do using POJOs or strings or something else inert (non-code). The functions won't be doing anything fancy anyway. All the "custom code" in those functions will ever do is call one of the predefined constructors.

[robertleeplummerjr]

@riatzukiza, ty for the feedback! @ddsol I really like where you are going with this as well. Do you by chance have any recommendations or pseudo code? I know you are doing some work on the rnn care to share sometime soon, as I believe it follows a similar path?

[ddsol]

One possibility would be for instance:

theCnn = new CNN({
  input: {
   size: 3 // <== only size is known, the actual input comes later. Maybe flexible size is possible instead?
  },
  hiddenLayers: [
    {
      type: Convolution
      depth: 1,
      filters: 6,
      padding: 1,
      stride: 1
    },
    Relu,
    {
      type: Pool
      padding: 1,
      stride: 1,
      width: 4,
      height: 4
    },
    40 //eventually backward compatible flat array, just for visuals (???)
  ] //output is implicit
}); // <== this compiles a NN and possibly optimizes it

theCnn.run([35, 35, 3]); // <== pass the datas after construction, possibly multiple times

I'm not currently working on any RNN or CNN or any other NN.

[robertleeplummerjr]

@ddsol thank you for the suggestion and clarification, I really like what you did there.

[robertleeplummerjr]

As a side note, the algorithms used (at least at this time) will be to compile a simple kernel, rather than to run all the math for performance. If interested in why, check this out: https://jsperf.com/loop-vs-loopless

While this isn't exactly the same thing and the size is substantially smaller, on my system I get 85 times faster results from a loopless or "compiled" algorithm. If we can achieve twice as fast as other current libraries, I'm happy.

[Dok11]

@robertleeplummerjr is any conv nn was published on brain.js or this only discuss?

[robertleeplummerjr]

It is partially implemented here https://github.com/harthur-org/brain.js/tree/convolutional/src. I'll commit what I have locally later today, which isn't working fully yet, but is substantially further than what is here.

[robertleeplummerjr]

I forgot to mention on the loop vs loopless, that 85 times faster is cpu only, not even gpu yet.

[robertleeplummerjr]

Here is the output of the not yet working convolutional kernel: https://gist.github.com/robertleeplummerjr/ea2a4cd06012ccbca47a8597522759c4

from https://github.com/harthur-org/brain.js/tree/convolutional

exciting stuff!

[niembro64]

was a CNN ever achieved here?

(i.e. a convolutional layer / pooling layer)

[RezaErfani67]

was a CNN ever achieved here?