dannyneil/spiking_relu_conversion

spiking_relu_conversion's Issues

Training using cifar10

@dannyneil Hi Dan,

Recently, I want to use the cifar10 datasets to training example_conv network. When I change part of the code for the cifar10, but there is a error I donnot know how to solve it.
%% Load data
rand('state', 1);
% load mnist_uint8;
% train_x = double(reshape(train_x',28,28,60000)) / 255;
% test_x = double(reshape(test_x',28,28,10000)) / 255;
% train_y = double(train_y');
% test_y = double(test_y');
%%%%%%%%%%%%cifar10%%%%%%%%%%%%%%%%%%%%
load data;
load labels;
train_x = double(reshape(data',32,32,3,50000)) / 255;
for i = 1:1:50000,
index1=labels(i)+1;
train_y(i,index1)=1;
end
train_y = double(train_y');

load test_batch;
test_x = double(reshape(data',32,32,3,10000)) / 255;
test_y = zeros(10000,10);
for i = 1:1:10000,
index2=labels(i)+1;
test_y(i,index2)=1;
end
test_y = double(test_y');
%%%%%%%%%%%%%%%%%%%%%
cnnff.m
%%%%%%%%%%%%%%%%%%%%%%%%
function net = cnnff(net, x)
n = numel(net.layers);
net.layers{1}.a{1} = x;
%disp(size(x));
unused_inputs = rand(size(x)) < net.first_layer_dropout;
net.layers{1}.a{1}(unused_inputs) = 0;
inputmaps = 3;%1

for l = 2 : n   %  for each layer
    if strcmp(net.layers{l}.type, 'c')
        for j = 1 : net.layers{l}.outputmaps   %  for each output map
            if net.layers{l}.used_maps(j)
                %  create temp output map
                z = zeros(size(net.layers{l - 1}.a{1}) - [net.layers{l}.kernelsize - 1 net.layers{l}.kernelsize - 1 0 0]);%%%%%0[28 28 3 50]
                %disp(size(z));
                for i = 1 : inputmaps   %  for each input map
                    %  convolve with corresponding kernel and add to temp output map
                    %xx=net.layers{2 - 1}.a{1};
                    %yy=net.layers{2}.k{1}{1};
                    %aa = convn(net.layers{l - 1}.a{i}, net.layers{l}.k{i}{j}, 'valid');
                    %disp(size(aa));
                    z = z + convn(net.layers{l - 1}.a{i}, net.layers{l}.k{i}{j}, 'valid');
                end
                %  add bias, pass through nonlinearity
                net.layers{l}.a{j} = net.act_fun(z + net.layers{l}.b{j});
            else
                net.layers{l}.a{j} = zeros(size(net.layers{l - 1}.a{1}) - [net.layers{l}.kernelsize - 1 net.layers{l}.kernelsize - 1 0 0]);
            end
        end
        %  set number of input maps to this layers number of outputmaps
        inputmaps = net.layers{l}.outputmaps;
    elseif strcmp(net.layers{l}.type, 's')
        %  downsample
        for j = 1 : inputmaps
            z = convn(net.layers{l - 1}.a{j}, ones(net.layers{l}.scale) / (net.layers{l}.scale ^ 2), 'valid');   %  !! replace with variable
            net.layers{l}.a{j} = z(1 : net.layers{l}.scale : end, 1 : net.layers{l}.scale : end, :);
        end
    end
end

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
The error is:

example_convnet
epoch 1/5
Index exceeds matrix dimensions.
Error in cnnff (line 22)
z = z + convn(net.layers{l - 1}.a{i}, net.layers{l}.k{i}{j}, 'valid');
Error in cnntrain (line 24)
net = cnnff(net, batch_x);
Error in example_convnet (line 52)
cnn = cnntrain(cnn, train_x, train_y, opts)

So could you give me some ideas to solve this problem. Thank you very much.

Bin

ReLU in backprop

Hi Dan!
Thanks for sharing your code. I was trying to add ReLU support to DeepLearningToolbox when I found this repository. Would you be so kind to help me with my questions?

1. I'm looking at the backprop realization for ReLU and can't understand one thing about handling ReLU at output layer, see line 11 at nnbp.m.
   Looks like you decided to set the derivative of Loss-function w.r.t. total input of ReLU-unit in output layer to the same formula, as expected for simple linear unit, ignoring the fact that ReLU's derivative is zero for negative input. Shouldn't the ReLU in output layer have a special handling, as it does in hidden layers? Something like

case 'relu'
    d{n} = - nn.e .* single(nn.a{n} > 0);

instead of

case {'softmax','linear', 'relu'}
        d{n} = - nn.e;

2. Anyway, with the help of your code, I added ReLU support to clean DeepLearningToolbox instance. Then I tried to test gradients for numerical correction by updating test_nn_gradients_are_numerically_correct() to include relu in tests. But then the test started to throw assertion violation in nnchecknumgrad() with 'relu' activation and output 'linear'. Mere raising error threshold in 10 times more didn't help so I wonder why is that? I checked the relu realization multiple times, but it look good (BTW: that's when I found the reason for the first question, but it doesn't matter in this question, because output is linear unit)
   Did you tried to update your version of test_nn_gradients_are_numerically_correct() to test ReLU?

3. Just curious, why did you remove bias units from NN code? Biases significantly enhances neuron ability to discriminate various patterns. What's the point to remove them?

Thanks!

Error showing "Index exceeds the number of array elements(4) in lmiunpck"

I have gone through your research work "Fast-Classifying, High-Accuracy Spiking DeepNetworks Through Weight and Threshold Balancing" and tried to do the simulation using the MATLAB code provided by you on github. But when I am trying to run it in my MATLAB 2021a,it is showing following errors :

" Index exceeds the number of array elements (4).
Error in lmiunpck (line 22)
rs=lmisys(4); rv=lmisys(5); % row sizes of LMISET,LMIVAR
Error in nnsetup (line 25)
[LMI_set,LMI_var,LMI_term,data]=lmiunpck(lmisys);
Error in example_fcn (line 12)
nn = nnsetup([784 1200 1200 10]);"

I want the same result as the paper.

Hi
I read your paper with interest.
but, I couldn't get the results of your paper experiment with this code.
The code below is the parameter values I used.

cnn.layers = {
struct('type', 'i') %input layer
struct('type', 'c', 'outputmaps', 12, 'kernelsize', 5) %convolution layer
struct('type', 's', 'scale', 2) %sub sampling layer
struct('type', 'c', 'outputmaps', 64, 'kernelsize', 5) %convolution layer
struct('type', 's', 'scale', 2) %subsampling layer
};
cnn = cnnsetup(cnn, train_x, train_y);
% Set the activation function to be a ReLU
cnn.act_fun = @(inp)max(0, inp);
% Set the derivative to be the binary derivative of a ReLU
cnn.d_act_fun = @(forward_act)double(forward_act>0);
%% ReLU Train
% Set up learning constants
opts.alpha = 1;
opts.batchsize = 50;
opts.numepochs = 50;
opts.learn_bias = 0;
opts.dropout = 0.5;
cnn.first_layer_dropout = 0;

How should I fix it?

"previous_factor" may has a error

The file "normalize_nn_data.m" and "normalize_cnn_data.m" have the code such as :
for l = 1 : numel(nn.size)-1
% Find the max and rescale
weight_max = max(max(max(0, nn.W{l})));
activation_max = max(max(max(0, nn.a{l+1})));
scale_factor = max(weight_max, activation_max);
applied_inv_factor = scale_factor / previous_factor;
nn.W{l} = nn.W{l} / applied_inv_factor;
factor_log(l) = 1 / applied_inv_factor;
previous_factor = applied_inv_factor;
end

or
for ii = 1 : numel(net.layers{l - 1}.a)
for j = 1 : numel(net.layers{l}.a)
net.layers{l}.k{ii}{j} = ...
net.layers{l}.k{ii}{j} / current_factor;
end
end
factor_log(l) = 1 / current_factor;
previous_factor = current_factor;

So,the previous_factor equals to the applied_inv_factor or current_factor, rather than the scale_factor as in the paper.

Low performance

Thanks for sharing your code. I tested it on my computer. I believe the paper claimed 99.1% test performance. But I tried three times and only got around 95% test error. Is this normal? Is there anyway to improve?

Weight normalization factors

Hi,
When I ran this code on the network cnn_99.14, I see that the last layer weight scaling factor is different from that in the paper. Could you please let me know if the scaling technique used for last layer was the same as everywhere else?

Thanks!