Float values as output

[hughperkins]

Float values as output

[gilliM]

When I add a SquareLossMaker instead of a SoftMaxLayer, I got a crash.
I'm using the python API

[hughperkins]

Can you provide the exact script you're trying please? (Also: what OS? what version of python?)

[gilliM]

OS: Mac Os 10.11.2
Python: 2.7.10

For the code, the output should be a single float number, so I use the SquareLossMaker layer. Taking the provided example with modification, it gives someting like this:

net = PyDeepCL.NeuralNet(cl, 1, 28) net.addLayer(PyDeepCL.NormalizationLayerMaker().translate(-0.5).scale(1 / 255.0)) net.addLayer(PyDeepCL.ConvolutionalMaker().numFilters(8).filterSize(5).padZeros().biased()) net.addLayer(PyDeepCL.PoolingMaker().poolingSize(2)) net.addLayer(PyDeepCL.FullyConnectedMaker().numPlanes(1).imageSize(1)) net.addLayer(PyDeepCL.SquareLossMaker())

The python console print these following messages:
created netLearner
statefultimer v0.7
forward try kernel 0
... not plausibly optimal, skipping
forward try kernel 1
... seems valid
ForwardAuto: kernel 1 10ms
forward try kernel 0
... not plausibly optimal, skipping
forward try kernel 1
... seems valid
ForwardAuto: kernel 1 0ms

[hughperkins]

Ah. What are you using as the targets? Targets will be different dimensions when using squared loss. Should match the output dimensions of the network.

[hughperkins]

(ie, can you provide your complete test script please? but ideally, as simple a script as possible ,that replicates the problem)

[hughperkins]

Ah, I looked a bit more closely at your network. Ok, so it has a single float output, sounds good.

[hughperkins]

The following runs ok for me. Not saying it's learning anything, but it's not crashing:

#!/usr/bin/python

from __future__ import print_function
import array
import PyDeepCL
import sys
print('imports done')

cl = PyDeepCL.DeepCL()
net = PyDeepCL.NeuralNet(cl)
sgd = PyDeepCL.SGD(cl, 0.002, 0)
sgd.setMomentum(0.0001)

net = PyDeepCL.NeuralNet(cl, 1, 28)
net.addLayer(PyDeepCL.NormalizationLayerMaker().translate(-0.5).scale(1 / 255.0))
net.addLayer(PyDeepCL.ConvolutionalMaker().numFilters(8).filterSize(5).padZeros().biased())
net.addLayer(PyDeepCL.PoolingMaker().poolingSize(2))
net.addLayer(PyDeepCL.FullyConnectedMaker().numPlanes(1).imageSize(1))
net.addLayer(PyDeepCL.SquareLossMaker())

print(net.asString())
N = 1280
batchSize = 128
planes = 1
size = 28
numEpochs = 30

images = array.array('f', [0] * (N*planes*size*size))
targets = array.array('f', [0] * N)

net.setBatchSize(batchSize)
for epoch in range(numEpochs):
    print('epoch', epoch)
    numRight = 0
    context = PyDeepCL.TrainingContext(epoch, 0)
    for batch in range(N // batchSize):
        sgd.train(
            net,
            context,
            images[batch * batchSize * planes * size * size:],
            targets[batch * batchSize:])

[hughperkins]

Make sure to change from sgd.trainFromLabels to sgd.train by the way. train expects float targets. trainFromLabels expects integer labels.

[gilliM]

I try to fit the MNIST data set in a regressor way (why not). Just to test the code. The output explode and gives nan from the 2 nd epoch.

It's not clear for me how to do the prediction once the network is train (I think it with the forward function, and the get output from last layer, but the sizes doesn't fit...)

#!/usr/bin/python
# -*- coding: utf-8 -*-

from __future__ import print_function
import array
import PyDeepCL
import numpy as np
print('imports done')

path = "/Users/gmilani/cnn_data/"
mnistFilePath = path + 't10k-images-idx3-ubyte'

cl = PyDeepCL.DeepCL()

net = PyDeepCL.NeuralNet(cl)
net.addLayer(PyDeepCL.InputLayerMaker().numPlanes(1).imageSize(28))
net.addLayer(PyDeepCL.NormalizationLayerMaker().translate(-0.5).scale(1 / 255.0))
net.addLayer(PyDeepCL.ConvolutionalMaker().numFilters(8).filterSize(5).padZeros().biased())
net.addLayer(PyDeepCL.ActivationMaker().relu())
net.addLayer(PyDeepCL.PoolingMaker().poolingSize(2))
net.addLayer(PyDeepCL.ConvolutionalMaker().numFilters(8).filterSize(5).padZeros().biased())
net.addLayer(PyDeepCL.ActivationMaker().relu())
net.addLayer(PyDeepCL.PoolingMaker().poolingSize(3))
net.addLayer(PyDeepCL.FullyConnectedMaker().numPlanes(150).imageSize(1).biased())
net.addLayer(PyDeepCL.ActivationMaker().tanh())
net.addLayer(PyDeepCL.FullyConnectedMaker().numPlanes(1).imageSize(1).biased())
net.addLayer(PyDeepCL.SquareLossMaker())

sgd = PyDeepCL.SGD(cl, 0.002, 0)
sgd.setMomentum(0.0001)

print(net.asString())
N = 1280
batchSize = 128
planes = 1
size = 28
numEpochs = 3

images = array.array('f', [0] * (N * planes * size * size))
targets = array.array('i', [0] * N)

PyDeepCL.GenericLoader.load(mnistFilePath, images, targets, 0, N)
images = array.array('f', images)
targets = array.array('f', np.array(targets))
np_targets = np.array(targets)
print(np_targets)
precision = np.mean(np.array(targets))

net.setBatchSize(batchSize)
for epoch in range(0, numEpochs):
print('epoch', epoch)
context = PyDeepCL.TrainingContext(epoch, 0)

for batch in range(N // batchSize):
    sgd.train(
        net,
        context,
        images[batch * batchSize * planes * size * size:],
        targets[batch * batchSize:])

    net.forward(images[batch * batchSize * planes * size * size:])
    # test new getLabels() method:
    if batch == 0:
        lastLayer = net.getLastLayer()
        predictions = lastLayer.getOutput()
        # precision = np.mean((np.array(predictions) - np_targets) ** 2)
        # print('precision : ', precision)
        print(predictions)

[gilliM]

I've just changed the parameters and it works well !

sgd = PyDeepCL.SGD(cl, 0.0002, 0)
sgd.setMomentum(0.000001)

[hughperkins]

Cool. So it's working ok?

[gilliM]

Yes, it's training quite well. You can close the issue. You may want to do an example for regressor with this try for the documentation (?).

[hughperkins]

Yes, it's training quite well. You can close the issue.

Ok, cool :-)