jeshraghian/snntorch

Deep and online learning with spiking neural networks in Python

snntorch's Issues

Clean .flake8

The .flake8 config currently excludes lots of hints and errors. The code-base should be cleaned with the standard Flake8 config.

I would be glad to help out. Reporting as a reminder.

snntorch-ipu crashing

• snntorch version: 0.5.3
• snntorch-ipu version: 0.5.18
• PopTorch version: 2.6.0
• PyTorch version: 1.10.0
• Python version: 3.8.10
• Operating System: Ubuntu 20.04

Description

I've been trying to train a model in an IPU environment using PopTorch and snntorch-ipu. Unfortunately, I always get a crash. It is unclear to me what exactly is going on, so hopefully someone knows.

What I Did

If I try to train my model with only snntorch-ipu installed, as recommended, I will always get an error message when importing/working with surrogates about "Missing Straight Through Estimator Custom Operation file".

/notebooks/dvsclf/network/net.py in <module>
      1 import torch
----> 2 from snntorch import surrogate
      3 from snntorch import utils
      4 import torch.nn as nn
      5 import numpy as np

/usr/local/lib/python3.8/dist-packages/snntorch/surrogate.py in <module>
     26 
     27 
---> 28 class StraightThroughEstimator:
     29     """
     30     Straight Through Estimator.

/usr/local/lib/python3.8/dist-packages/snntorch/surrogate.py in StraightThroughEstimator()
     53         print("Missing Straight Through Estimator Custom Operation file!")
     54         print(so_path_ste)
---> 55         exit(1)
     56     ctypes.cdll.LoadLibrary(so_path_ste)
     57 

NameError: name 'exit' is not defined

If I install snntorch (with or without snntorch-ipu beside it), I will not get the above error. Instead, something in PopTorch throws an error when the model is being trained.

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-4-9ee252bfabe9> in <module>
      2     # Performs forward pass, loss function evaluation,
      3     # backward pass and weight update in one go on the device.
----> 4     _, loss = poptorch_model(batch, target)

[....]

/notebooks/dvsclf/network/snn.py in forward(self, x)
     20 
     21         x = self.conv(x)
---> 22         x = self.lif(x)
     23         return x
     24 

/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1118             input = bw_hook.setup_input_hook(input)
   1119 
-> 1120         result = forward_call(*input, **kwargs)
   1121         if _global_forward_hooks or self._forward_hooks:
   1122             for hook in (*_global_forward_hooks.values(), *self._forward_hooks.values()):

/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py in _slow_forward(self, *input, **kwargs)
   1088                 recording_scopes = False
   1089         try:
-> 1090             result = self.forward(*input, **kwargs)
   1091         finally:
   1092             if recording_scopes:

/usr/local/lib/python3.8/dist-packages/snntorch/_neurons/leaky.py in forward(self, input_, mem)
    159         if self.init_hidden:
    160             self._leaky_forward_cases(mem)
--> 161             self.reset = self.mem_reset(self.mem)
    162             self.mem = self.state_fn(input_)
    163 

/usr/local/lib/python3.8/dist-packages/snntorch/_neurons/neurons.py in mem_reset(self, mem)
     86         """Generates detached reset signal if mem > threshold.
     87         Returns reset."""
---> 88         mem_shift = mem - self.threshold
     89         reset = self.spike_grad(mem_shift).clone().detach()
     90 

/usr/local/lib/python3.8/dist-packages/poptorch/_poplar_executor.py in __torch_function__(cls, func, types, args, kwargs)
    279                     if kwargs is None:
    280                         kwargs = {}
--> 281                     return super().__torch_function__(func, types, args,
    282                                                       kwargs)
    283 

RuntimeError: Cannot insert a Tensor that requires grad as a constant. Consider making it a parameter or input, or detaching the gradient
Tensor:
(1,1,.,.) = 
 Columns 1 to 9  0.4529  0.4529  0.4529  0.4529  0.4529  0.4529  0.4529  0.4529  0.4529 ....

With self.lif = snn.Leaky(beta=0.95, spike_grad=surrogate.fast_sigmoid(), init_hidden=True, learn_beta=True, learn_threshold=True)

Tests: BatchNormThroughTime

Add new tests for BNTT1d and 2d.

Accuracy not improving on CIFAR-10 dataset

Hello!

I am trying out tutorial_6 (https://colab.research.google.com/github/jeshraghian/snntorch/blob/master/examples/tutorial_6_CNN.ipynb) but the only difference is I am using CIFAR-10 dataset instead of MNIST. I trained it for 10 epochs but the accuracy wasn't good, so I tried training for 50 epochs and the accuracy is still around 35-40 %. Can anyone please guide me on resolving this issue ?

Issue with inputing custom weights for Rate based SNN

• snntorch version: 0.4.4
• Python version: 3.9.6
• Operating System: Ubuntu 20.04.3 LTS

Description

Hi Jason,
First of all, I appreciate your wonderful effort in developing this package and a detailed documentation. I have recently started using snntorch for rate based SNN coding. Although I am getting good performance for purely software based run, I am facing issues with inputing custom weights extracted from a synaptic device. My accuracy is getting stuck at around 10% which is the same as the untrained accuracy.

What I Did

I used a custom function to input the weights from a text file as shown in the screenshot. Please let me know how to solve this issue.
NB: I am pretty new to programming. so pls excuse me if my code is too cumbersome :)

`
Here is the full file and the text file for data input
rate_SNN_dev_weights.zip

Thanks,
Kannan

spikegen.rate & spikegen.rate_conv: output dims don't match

• snntorch version: 0.2.6
• Python version: 3.6-3.8

Description

spikegen.rate and spikegen.rate_conv do not return equal dimensions.
One unsqueezes the output in the first dim.

What I Did

spk_in = spikegen.rate_conv(torch.rand((200, 784)))

The examples in Colab is out of date

Description

The tutorial_1 code in Colab looks like based on an out-of-date snntorch version.

For example, the code of rate encoding in Colab is

spike_data, spike_targets = spikegen.rate(data_it, targets_it, num_outputs=num_outputs, num_steps=num_steps,
                                          gain=1, offset=0, one_hot=False, time_varying_targets=False)

While the up-to-date code in Github is

spike_data = spikegen.rate(data_it, num_steps=num_steps, gain=1, offset=0). 

As I tried, both from this link https://snntorch.readthedocs.io/en/latest/examples.html and clicking the open in colab icon will lead to the wrong version tutorials.

Solution

A quick solution is directly cloning the repository and running locally or loading the notebook to Colab via Github link according to https://colab.research.google.com/github/googlecolab/colabtools/blob/master/notebooks/colab-github-demo.ipynb#:~:text=Colab%20can%20load%20public%20github,with%20no%20required%20authorization%20step.

Firing inhibition only works for flattened layers

Description

the inhibition argument of neurons is only setup for single dimensional channels / outputs of nn.Linear
It does not work for convolutional layers.

The way I see it, there are two options:

1. Only allow 1 neuron to fire from an entire layer; this is how it is currently setup for Linear layers
2. Allow 1 neuron to fire per-channel. Trickier, as the channel dimension must be known.

'snntorch' has no attribute 'FastSigmoid'

• snntorch version: 0.2.8
• Python version: 2.7.5
• Operating System: Windows 10

Description

I tried to use the surrogate gradient function by running the following line of code:

spike_grad = snn.FastSigmoid.apply

However, I am getting an error stating that "module 'snntorch' has no attribute 'FastSigmoid'".

I see that it is not implemented yet to the neuron models in your init.py file, may I ask when the push is going to be done?

Moreover, I see that we have only examples of applying rate encoding to the MNIST dataset, I need to ask if it is possible to have an example showing how to use the Latency encoding may also be on a Convolution network?

Thanks in advance.

How to install Popular SDK

• snntorch version: Not needed
• Python version: Not needed
• Operating System: Not needed

Description

I would like to install snntorch. I found it requires poptorch, which is dependent on Popular SDK.
How can I download Popular SDK? GraphCore offers Popular SDK only for signed-up users.
For this reason, I cannot download Popular SDK... Please help me.

Any general tips to convert ANNs to SNNs?

I am developing an SNN model to substitute ANNs with SNNs. I have observed that the performance of SNNs cannot catch up that of ANNs. To make SNNs comparable to ANNs, I tried the following:

1. Use more units in each layers to ensure high activity that possibly occurs at ANNs.
2. Use learning rates smaller than those used for ANNs.

I found that these a bit contribute to the performance of SNNs.

Do you have any other tips to convert ANNs to SNNs when it comes to setting hyperparameters?

Enable custom neuron generators

Enable user to pass in state equations, to specify the state variable(s), and custom neuron class is automatically generated which can be treated like any other LIF neuron.

Does snntorch need to encode the input data into a spike train?

• snntorch version:0.4.10
• Python version:3.7
• Operating System:Win 11

Description

Hi Jason,
I didn't find the program code to encode the input data in tutorial 5, nor in snn.Leaky. Does this mean that the user can input data into the model in Tutorial 5 without coding the input data into spike train?

Initial Update

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

training datasets of variable time steps

• currently, the initialization of state variables step is only equipped for one common time step

possible solutions:

• modifying the init step to adjust to the input first dimension
• zero-extension along the time axis (sloppy fix, and might need a warning displayed to inform the user). Might be best for user to implement this themselves. Thoughts are welcome.

Enable stride != 1 for SConv2dLSTM

SConv2dLSTM currently concatenates the hidden cell with the input cell, applies a single nn.Conv2d function of channel length 4*out_channel
The output is then split into 4 chunks (along the channel dimension), which are fed to each of the 4 LSTM gates
This means HxW of the hidden cell must be equal to HxW of input

Stride != 1 causes HxW to no longer be equal.
To fix this, a stride argument can trigger the conv to take place before concatenation of the hidden state mem and input x.
Far less efficient than running all conv operations in one hit, though I don't yet see a way around this.

Snntorch tutorials

About using NMNIST dataset, num_steps=300, dt=1000

Good day!

I wanted to know what does mean num_steps = 300. Does it mean that to use only 300 events?
Like we have 60000 samples in NMNIST, and each sample has around 4500 events. But if we set dt =1000,
and num_steps = 300, so we use only 300 events out of 4500?

Thank you!

How to reflect the consumption gap between SNN and CNN

What parameters are provided to compare the difference in energy consumption between the two, such as the amount of computation?

RSynaptic Neuron Model NOT in snnTorch

Hello,
I am trying to use the RSynaptic neuron model but even though it is documented in the snnTorch website, the class RSynaptic(LIF) is not included in the snntroch module.
This this the error I get when I try to use it:

AttributeError: module 'snntorch' has no attribute 'RSynaptic'

Axonal Delays

• Add gated spiking units with learnable delay times
• Integrate it with each LIF neuron

Lapicque model needs tests

• snntorch version: 0.2.4
• Python version: 3.6-3.8

Description

No unit tests for the Lapicque neuron model.

What I Did

No errors thrown - just missing feature.

Regression Loss

• snntorch version: lateset
• python 3.8:
• Ubuntu18:

Description

Is snntorch loss suitable for regression loss function? Because I found that snn loss seems only suitable for classification loss.

homeostasis

Add an adaptive, time-varying threshold to all neuron models.

• threshold should decay back to steady-state value over some time-constant

• threshold may need to saturate to some value to avoid removing firing activity

• run tests to determine if it should be detached from the computational graph (likely to be the case, as it will only be activated upon spike firing)

`backprop.BPTT` cannot handle variable `num_steps` across minibatches

Getting around this should be quite straightforward:

• Convert num_steps into an optional argument (but an error is raised if input data is time-series)
• In the foward-loop, rather than using num_steps, instead use data.size(0)

Discussed in #74

^{Originally posted by msbouanane November 26, 2021}
Hello,
The backprop.BPTT function requires the num_steps parameter. But when using a neuromorphic dataset, the num_steps will be variable depending on the longest recording in the mini-batch.
What should I pass to this parameter instead?

AttributeError: module 'snntorch' has no attribute 'spikegen'

• snntorch version: 0.5.1
• Python version: 3.7
• Operating System: Windows

Description

When I try to reproduce one of the example, e.g.

import snntorch as snn

a = torch.Tensor([1, 2, 2.9, 3, 3.9])
snn.spikegen.delta(a, threshold=1)

I get the error module 'snntorch' has no attribute 'spikegen'. Where do I find the module?

TBPTT mode gives error about K parameter on time varying data

• snntorch version: 0.5.3
• Python version: 3.8
• Operating System: windows

when i try to use TBPTT or RTRL from backprop on time varying signal, it gives me this error:

Java Printing.pdf

    if K_flag is False:
UnboundLocalError: local variable 'K_flag' referenced before assignment

latency() got an unexpected keyword argument 'num_outputs' and latency() got multiple values for argument 'num_steps'

• snntorch version: 0.2.11
• Python version: 3.7.10

Description

In the latest updates, I see that the num_output has been removed from the spikegen. No clue why!

Moreover, also when I just decide about removing the num_output parameter. I get another error from the num_steps when I set it to any int value?

BTW, the same exists if I tried to do rate encoding instead of latency encoding.

What I Did

Here: you will find a quick ipynb file that shows the errors.

spikevision: neuromorphic dataloaders

automate dataset download/extraction/transform process in module spikevision.

• NMNIST
• DVS Gesture
• SHD

learnable time constants

• a global learnable time constant was implemented in 'learnable-decay'
• extending this to layer-wise time constants, and potentially neuronal independent time constants might yield more promising results
• bonus points: testing the best initialization of learnable-decay + integrating them as class methods within each neuron class

URL Fix in tutorial 6

• snntorch version: 0.4.11
• Python version: 3.9
• Operating System: Windows 11

Description

While I am looking through your tutorials, I have noticed that in tutorial 6, the following line:

Part of this tutorial was inspired by Friedemann Zenke’s extensive work on SNNs. Check out his repo on surrogate gradients here, and a favourite paper...

seems to have an unresolved URL https://github.corm/fzenke/spytorch, which I think 'corm' is misspelled.

What I Did

Click the link.

How is energy consumption reflected in the comparison between SNN and CNN?

What is the power efficiency between the two, the amount of computation or some other parameter

Neurons can fire multiple time steps in a row.

It is possible to have neurons firing continuously using the default reset mechanism or if using RLeaky and reset to zero. The latter is due to only resetting the input but not the recurrent connections. This is undesirable behavior as it allows the neurons to essentially not be spiking neurons given the right weight values.

Minor Update to Dependencies list in README.rst

• snntorch version: 0.5.3
• Python version: 3.10.4
• Operating System: Windows 10

Description

I think that ffmpeg is missing from the dependencies listed in README.rst.

What I Did

Using conda, I installed the dependencies and then received the following error on the video conversion step of the first tutorial:

RuntimeError: Requested MovieWriter (ffmpeg) not available

This was easily sorted with: conda install -c conda-forge ffmpeg.

Citation from README incorrect code

Description

The citation included in the README file has an incorrect reference:

@article{eshraghian2021training,
title={Training spiking neural networks using lessons from deep learning},
author={Eshraghian, Jason K and Ward, Max and Neftci, Emre and Wang, Xinxin
and Lenz, Gregor and Dwivedi, Girish and Bennamoun, Mohammed and Jeong, Doo Seok
and Lu, Wei D},
journal={arXiv preprint arXiv:1906.09395},
year={2021}
}

What I Did

I assume the correct arXiv code should be: 2109.12894:

@article{eshraghian2021training,
title={Training spiking neural networks using lessons from deep learning},
author={Eshraghian, Jason K and Ward, Max and Neftci, Emre and Wang, Xinxin
and Lenz, Gregor and Dwivedi, Girish and Bennamoun, Mohammed and Jeong, Doo Seok
and Lu, Wei D},
journal={arXiv preprint arXiv:2109.12894},
year={2021}
}

Spike Timing loss/accuracy functions in snn.functional

Other than fucntional.mse_membrane_loss(), all other loss functions available are only able to train rate-coded networks.

Temporal / time-to-first-spike mechanisms are more challenging as they involve a conversion from the spike (1/0) domain to the time domain. The most obvious fix is to simply iterate through the output spike tensor's time-axis, and populate a new tensor with the time at which each spike occurs. This would then be passed into a MSELoss function.

But indexing breaks the computational graph in PyTorch, so this would have to be overridden using a pre-defined backward function that is inherited from torch.autograd.Function.

Bonus features:

• tolerance for spike-timing
• relative spike-timing
• account for all spikes, not just the first

Tutorial 6 Net code structure issue

• Operating System: Colab

Description

While implementing the LeakySigmoidSurrogate model presented in tutorial 6 I came across different outputs from my personal implementation following the Colab Notebook and snn.Leaky() with the same configurations.

After diving into snn.Leaky() implementation I realised that the Net forward function is not the same.

• Current forward

    spk = self.surrogate_func((mem - self.threshold), self.slope)  # call the Heaviside function
    reset = (spk - self.threshold).clone().detach()
    mem = self.beta * mem + input_ - reset
  

• snn.Leaky() forward

    mem_shift = mem - self.threshold
    reset = self.surrogate_func(mem_shift, self.slope).clone().detach()
    mem = self.beta * (mem - reset * self.threshold) + input_
    spk = self.surrogate_func(mem_shift, self.slope)  # call the Heaviside function
  

As well I think that in the init instead of super(Leaky_Surrogate, self).__init__() should be super(LeakySigmoidSurrogate, self).__init__()

a more intuitive approach to handling targets

right now, spikegen.rate, spikegen.latency and spikegen.delta accept targets as an input.
But there is no way to specify how to encode the targets beyond one-hot-encodings & time-varying data.

• perhaps removing target conversion as an option if it is redundant. spike-encoded targets should be passed into a separate instance of spikegen.rate or spikegen.latency.
• Additional arguments for targets to set a minimum and maximum values for rate/latency codes. This would be useful for applying MSELoss with membrane potential.
• Tutorials & tests must be updated to handle these changes.

Question on loss of temporal coding.

Hi,
I 'm doing some latency code training with snntorch. I'm not very sure how to calculate the loss of temporal coding. In rate coding, we just sum the loss of ervery time steps, but how to calculate the loss in temporal coding?

Thanks.

Enable custom surrogate gradient functions

Enable user to pass in an arbitrary function which is automatically centered about the firing threshold.

Mixed-mode autodiff for RTRL

• Implement forward-mode AD for temporal connections; reverse-mode AD for layer connections
• Update at each time step
• Approximate BPTT without having to store any historical gradients

snntorch multi GPU training issue

• snntorch version: snntorch: 0.5.3
• Python version: 3.9
• Operating System: linux
• nvidia-smi

Every 0.5s: nvidia-smi                                                                                                                                    neuro: Fri Dec  2 11:16:53 2022

Fri Dec  2 11:16:53 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.82.01    Driver Version: 470.82.01    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA RTX A6000    On   | 00000000:1B:00.0 Off |                  Off |
| 30%   30C    P8    29W / 300W |      1MiB / 48682MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA RTX A6000    On   | 00000000:1C:00.0 Off |                  Off |
| 30%   27C    P8    22W / 300W |      1MiB / 48685MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  NVIDIA RTX A6000    On   | 00000000:1D:00.0 Off |                  Off |
| 30%   32C    P8    23W / 300W |      1MiB / 48685MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  NVIDIA RTX A6000    On   | 00000000:1E:00.0 Off |                  Off |
| 30%   31C    P8    23W / 300W |      1MiB / 48685MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   4  NVIDIA RTX A6000    On   | 00000000:3D:00.0 Off |                  Off |
| 30%   27C    P8    22W / 300W |      1MiB / 48685MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   5  NVIDIA RTX A6000    On   | 00000000:3F:00.0 Off |                  Off |
| 30%   29C    P8    23W / 300W |      1MiB / 48685MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   6  NVIDIA RTX A6000    On   | 00000000:40:00.0 Off |                  Off |
| 30%   27C    P8    22W / 300W |      1MiB / 48685MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   7  NVIDIA RTX A6000    On   | 00000000:41:00.0 Off |                  Off |
| 30%   30C    P8    22W / 300W |      1MiB / 48685MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Description

I'm trying to train NMNIST with snntorch using multi GPU. since snntorch is based on torch package, I thought data parrallel from torch nn should work.
here's whole code.

import torch
import torchvision.datasets as dsets
import torchvision.transforms as transforms
import torch.nn.init
import os
import torch.nn as nn
import time
import matplotlib.pyplot as plt
import tonic.transforms as transforms
import tonic
import numpy as np
import snntorch as snn
from snntorch import surrogate
from snntorch import functional as SF
from snntorch import spikeplot as splt
from snntorch import utils
import torch.nn as nn
import os
from torch.utils.data import DataLoader, random_split
import torch
device = 'cuda' if torch.cuda.is_available() else 'cpu'
sensor_size = tonic.datasets.NMNIST.sensor_size

# Denoise removes isolated, one-off events
# time_window
frame_transform = transforms.ToFrame(sensor_size=sensor_size, time_window=1)


frame_transform = transforms.Compose([transforms.Denoise(filter_time=10000),
                                      transforms.ToFrame(sensor_size=sensor_size,
                                                         time_window=50000)
                                     ])

trainset = tonic.datasets.NMNIST(save_to='/home/hubo1024/PycharmProjects/snntorch/data/NMNIST', transform=frame_transform, train=True)
testset = tonic.datasets.NMNIST(save_to='./home/hubo1024/PycharmProjects/snntorch/data/NMNIST', transform=frame_transform, train=False)

# seed fix
torch.manual_seed(777)

# seed fix if gpu is available
if device == 'cuda':
    torch.cuda.manual_seed_all(777)

#batch_size = 100

batch_size = 32
dataset_size = len(trainset)
train_size = int(dataset_size * 0.9)
validation_size = int(dataset_size * 0.1)


trainset, valset = random_split(trainset, [train_size, validation_size])
print(len(valset))
print(len(trainset))
trainloader = DataLoader(trainset, batch_size=batch_size, collate_fn=tonic.collation.PadTensors(), shuffle=True)
valloader = DataLoader(valset, batch_size=batch_size, collate_fn=tonic.collation.PadTensors(), shuffle=True)
testloader = DataLoader(testset, batch_size=batch_size, collate_fn=tonic.collation.PadTensors())


spike_grad = surrogate.fast_sigmoid(slope=75)
beta = 0.5

class CNN(torch.nn.Module):

    def __init__(self):
        super(CNN, self).__init__()
        self.keep_prob = 0.5
        self.layer1 = torch.nn.Sequential(
            nn.Conv2d(2, 12, 5),
            nn.MaxPool2d(2),
            snn.Leaky(beta=beta, spike_grad=spike_grad, init_hidden=True)
        )

        self.layer2 = torch.nn.Sequential(
            nn.Conv2d(12, 32, 5),
            nn.MaxPool2d(2),
            snn.Leaky(beta=beta, spike_grad=spike_grad, init_hidden=True)
        )

        self.layer4 = torch.nn.Sequential(
            nn.Flatten(),
            nn.Linear(32 * 5 * 5, 10),
            snn.Leaky(beta=beta, spike_grad=spike_grad, init_hidden=True, output=True)
        )

    def forward(self, data):
        spk_rec = []
        layer1_rec = []
        layer2_rec = []
        utils.reset(self.layer1)  # resets hidden states for all LIF neurons in net
        utils.reset(self.layer2)
        utils.reset(self.layer4)

        for step in range(data.size(1)):  # data.size(0) = number of time steps
            input_torch = data[:, step, :, :, :]
            input_torch = input_torch.cuda()
            #print(input_torch)
            out = self.layer1(input_torch)
            #out1 = out

            out = self.layer2(out)
            #out2 = out
            out, mem = self.layer4(out)
            #out = self.layer4(out)

            spk_rec.append(out)

            #layer1_rec.append(out1)
            #layer2_rec.append(out2)

        return torch.stack(spk_rec)#, torch.stack(layer1_rec), torch.stack(layer2_rec)


model = CNN().to(device)
device_ids = [0, 1] #your GPU index
model = torch.nn.DataParallel(model, device_ids=device_ids)
#model = nn.DataParallel(model).to(device)
optimizer = torch.optim.NAdam(model.parameters(), lr=0.005,betas=(0.9, 0.999))
loss_fn = SF.mse_count_loss(correct_rate=0.8, incorrect_rate=0.2)
#model = nn.DataParallel(model)

total_batch = len(trainloader)
print('총 배치의 수 : {}'.format(total_batch))
loss_fn = SF.mse_count_loss(correct_rate=0.8, incorrect_rate=0.2)
num_epochs = 15
loss_hist = []
acc_hist = []
v_acc_hist = []
t_spk_rec_sum = []
start = time.time()
val_cnt = 0
v_acc_sum= 0
avg_loss = 0
index = 0
#################################################


for epoch in range(num_epochs):
    torch.save(model.state_dict(), '/home/hubo1024/PycharmProjects/snntorch/model_pt/Radam_15epoch-50000.pt')
    for i, (data, targets) in enumerate(iter(trainloader)):
        data = data.cuda()
        targets = targets.cuda()
        model.train()

        spk_rec = model(data)

        #print(spk_rec.shape)
        loss_val = loss_fn(spk_rec, targets)
        avg_loss += loss_val.item()
        optimizer.zero_grad()

        loss_val.backward()

        optimizer.step()

        # Store loss history for future plotting
        loss_hist.append(loss_val.item())
        val_cnt = val_cnt+1
        #del loss_val


        if val_cnt == len(trainloader)/2-1:
            val_cnt=0

            for ii, (v_data, v_targets) in enumerate(iter(valloader)):
                v_data = v_data.to(device)
                v_targets = v_targets.to(device)

                v_spk_rec = model(v_data)
                #
                # print(t_spk_rec.shape)
                v_acc = SF.accuracy_rate(v_spk_rec, v_targets)
                del v_spk_rec
                if ii == 0:
                    v_acc_sum = v_acc
                    cnt = 1

                else:
                    v_acc_sum += v_acc
                    cnt += 1
                #del v_acc


            plt.plot(acc_hist)
            plt.plot(v_acc_hist)
            plt.legend(['train accuracy', 'validation accuracy'])
            plt.title("Train, Validation Accuracy-Radam 15epoch-50000")
            plt.xlabel("Iteration")
            plt.ylabel("Accuracy")
            # plt.show()
            plt.savefig('Radam_15epoch-50000.png')
            plt.clf()
            v_acc_sum = v_acc_sum/cnt


            # avg_loss = avg_loss / (len(trainloader) / 2)
            # print('average loss while half epoch', avg_loss)
            # if avg_loss <= 0.5:
            #     index = 1
            #     break
            # else:
            #     avg_loss = 0
            #     index = 0

        print('Radam-15epoch-50000')
        print("time :", time.time() - start,"sec")
        print(f"Epoch {epoch}, Iteration {i} \nTrain Loss: {loss_val.item():.2f}")

        acc = SF.accuracy_rate(spk_rec, targets)
        acc_hist.append(acc)
        v_acc_hist.append(v_acc_sum)
        print(f"Train Accuracy: {acc * 100:.2f}%")
        print(f"Validation Accuracy: {v_acc_sum * 100:.2f}%\n")

    #     if index == 1:
    #         break
    # if index == 1:
    #     break
# 학습을 진행하지 않을 것이므로 torch.no_grad()
'''
with torch.no_grad():
    X_test = mnist_test.test_data.view(len(mnist_test), 1, 28, 28).float().to(device)
    Y_test = mnist_test.test_labels.to(device)

    prediction = model(X_test)
    correct_prediction = torch.argmax(prediction, 1) == Y_test
    accuracy = correct_prediction.float().mean()
    print('Accuracy:', accuracy.item())
'''

and here's error

(snn_torch) hubo1024@neuro:~/PycharmProjects/snntorch$ CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0,1,2,3,4,5 python gpu_6_run.py
6000
54000
총 배치의 수 : 13500
Traceback (most recent call last):
  File "/home/hubo1024/PycharmProjects/snntorch/gpu_6_run.py", line 146, in <module>
    spk_rec = model(data)
  File "/home/hubo1024/anaconda3/envs/snn_torch/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/hubo1024/anaconda3/envs/snn_torch/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py", line 168, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/home/hubo1024/anaconda3/envs/snn_torch/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py", line 178, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/home/hubo1024/anaconda3/envs/snn_torch/lib/python3.9/site-packages/torch/nn/parallel/parallel_apply.py", line 86, in parallel_apply
    output.reraise()
  File "/home/hubo1024/anaconda3/envs/snn_torch/lib/python3.9/site-packages/torch/_utils.py", line 461, in reraise
    raise exception
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/home/hubo1024/anaconda3/envs/snn_torch/lib/python3.9/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/home/hubo1024/anaconda3/envs/snn_torch/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/hubo1024/PycharmProjects/snntorch/gpu_6_run.py", line 102, in forward
    out = self.layerconv1(input_torch)
  File "/home/hubo1024/anaconda3/envs/snn_torch/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/hubo1024/anaconda3/envs/snn_torch/lib/python3.9/site-packages/torch/nn/modules/container.py", line 139, in forward
    input = module(input)
  File "/home/hubo1024/anaconda3/envs/snn_torch/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/hubo1024/anaconda3/envs/snn_torch/lib/python3.9/site-packages/snntorch/_neurons/leaky.py", line 162, in forward
    self.mem = self.state_fn(input_)
  File "/home/hubo1024/anaconda3/envs/snn_torch/lib/python3.9/site-packages/snntorch/_neurons/leaky.py", line 201, in _build_state_function_hidden
    self._base_state_function_hidden(input_) - self.reset * self.threshold
  File "/home/hubo1024/anaconda3/envs/snn_torch/lib/python3.9/site-packages/snntorch/_neurons/leaky.py", line 195, in _base_state_function_hidden
    base_fn = self.beta.clamp(0, 1) * self.mem + input_
  File "/home/hubo1024/anaconda3/envs/snn_torch/lib/python3.9/site-packages/torch/_tensor.py", line 1121, in __torch_function__
    ret = func(*args, **kwargs)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

I rerun this code after removing snn.Leaky layer in CNN and it worked fine. (of course the cost doesn't converge and accuracy was 0% but still it runs)
So I assume that the reason of this error is snn.Leaky layer.
I think changing

Examples of regression?

I was wondering if anyone had used snnTorch for regression, and perhaps how you set your networks up. Just looking for simple, general examples! MSELoss would likely be the type of loss used as I see it.

latency() got an unexpected keyword argument 'num_outputs'

• snntorch version:
• Python version:
• Operating System:

Description

Describe what you were trying to get done.
Tell us what happened, what went wrong, and what you expected to happen.

What I Did

Paste the command(s) you ran and the output.
If there was a crash, please include the traceback here.

Add class imbalance weighting to loss functions

Apply on/off target weighting to snntorch.functional losses in the same way the PyTorch enables weighting.

Cross Entropy-based losses should be straightforward; Mean Square Error Losses are less trivial.

Add power profiling capabilities

This seems to be a super popular feature request.
Making accurate estimates seems near impossible, but we can probably generate an order of magnitude guess here.

The user would construct a model, pass data in, and the power profiling function returns the number of Synaptic operations in the forward-pass (this could be averaged across batches).

Each synaptic op would be scaled by the energy cost for all selected devices; e.g., various GPUs & neuromorphic hardware.
The same number would be given for non-spiking networks too. This could be achieved by just removing the spiking modules.

SpikingKeras has a similar function that does it really nicely. However, it overstates the improvement given with spikes because it does not account for overhead (i.e., moving data to/from memory, or between multiple chips).

Including an argument that factors in overhead would by tricky, but useful.
The model would be parsed for number of neurons/synapses, and if either exceeds the bandwidth of a single chip, then we need to estimate how frequently data needs to be moved between chips & add that to the overall energy consumption.

A lot of coarse estimates would be made, but I think it could be helpful.

Networks Learns Nothing!

• snntorch version: 0.2.8
• Python version: 3.7.4
• Operating System: Windows 10

Description

Hi,

I tried to use the snnTorch to do exactly as you are doing in Tutorial 3 (without spike_grad), and also in the upcoming Tutorial 4 (applied spike_grad to stein neuron) for Spiking CNNs. Moreover, I also converted my dataset from static to spike version as in Tutorial 4 using rate encoding.

However, usually, I found that the network learns nothing and the training loss goes down and up slightly, but the output numbers and accuracy are completely unintuitive.

For example, I used a Conventional CNN on my Dataset and I got 85% accuracy, and I used it with the same architecture and hyperparameters and nothing is learned.

I will be glad if you can have some direct suggestions about what may be the problem and why the network is learning nothing in the Spiking domain?

Looking forward to your response

Tutorial 7 - event_tensor shape is unmatched

• snntorch version: 0.5.3
• Python version: 3.8.10
• Operating System: Ubuntu 20.04

Description

At "Tutorial 7 - NEUROMORPHIC DATASETS WITH TONIC + SNNTORCH", shape of event_tensor, made by torch.utils.data.DataLoader, is depicted as torch.Size([311, 128, 2, 34, 34]). But when I run the codes, shape of event_tensor shown as torch.Size([128, 311, 2, 34, 34]), first and second dimensions are changed. So I couldn't train anything, because tensor shape is not matched exactly.

Why does the shape of tutorial look different from the actual implementation?

Was there any update of snnTorch or torch(DataLoader)?

What I Did

Here is colab error message. I didn't modify anything, just ran tutorial code.

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/loss.py:530: UserWarning: Using a target size (torch.Size([128, 10])) that is different to the input size (torch.Size([312, 10])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  return F.mse_loss(input, target, reduction=self.reduction)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
[<ipython-input-18-5a0ce5857296>](https://localhost:8080/#) in <module>
     13         net.train()
     14         spk_rec = forward_pass(net, data)
---> 15         loss_val = loss_fn(spk_rec, targets)
     16 
     17         # Gradient calculation + weight update

4 frames
[/usr/local/lib/python3.7/dist-packages/torch/functional.py](https://localhost:8080/#) in broadcast_tensors(*tensors)
     71     if has_torch_function(tensors):
     72         return handle_torch_function(broadcast_tensors, tensors, *tensors)
---> 73     return _VF.broadcast_tensors(tensors)  # type: ignore[attr-defined]
     74 
     75 

RuntimeError: The size of tensor a (312) must match the size of tensor b (128) at non-singleton dimension 0

Recurrent Leaky & Synaptic Neurons - expand 1-1 connection

Issue: The present implementation of RLeaky and RSynaptic each assign only a single weight per neuron in the recurrent feedback connection.

Proposed Fix: This should be modified to account for cases where the linear connection should be fed through a parameterizable dense (and possibly conv?) recurrent connection.

Possible implementation:

lif1 = snn.RLeaky(beta=beta, dense=128) # each output spike is looped back to the input via a dense layer of 128 neurons

Challenge with above implementation:

This cannot be applied to convolutional layers. The shape of the output tensor must be projected back to whatever the input tensor is such that the two can be added or appended. (Q: which one?)

For convs, perhaps the input data has a 'size check' to infer how padding, stride, kernel size must be applied.
Ideally, the user could specify kernel size and everything else is inferred (e.g., depth).

We already do something similar for the spiking Conv2d LSTM neuron, too.

A problem occurred when running the code in tutorial 2

• snntorch version:0.4.4
• Python version:3.8
• Operating System:Windows 10

Description

Hello Jason,
When I ran the code in Tutorial 2, I found that several ploting functions were undefined. Can you tell me which package they are included in so that I can install the missing package？
Looking forward to your reply.

Thanks,
Sasuke

What I Did