zk-zhou/spikformer

ICLR 2023, Spikformer: When Spiking Neural Network Meets Transformer

• 👋 Hi, I’m @ZK-Zhou
• 👀 I’m interested in machine learning.
• 🌱 I’m currently learning math.
• 💞️ I’m looking to collaborate on publishing paper.
• 📫 How to reach me ...

spikformer's Issues

Error of "enumerate(loader) " in line 647 of "imagenet/train.py"

在 "train.py"的647行，需要 "enumerate(loader)"来遍历取出“batch_idx,(input, target)两个变量。
但是会出现timm/data/transforms.py报错”line 18, in __call__np_img = np.array(pil_img, dtype=np.uint8)“，”TypeError: array() takes 1 positional argument but 2 were given“。

不知道是不是timm的版本库问题或是loader数据失败。其中，timm为0.5.4， "train.py"647行处断点”loader“的数据如下：

result<timm.data.loader.PrefetchLoader object at 0x7f8f79a7d100>
dataset: <timm.data.dataset.ImageDataset object at 0x7f8ffeb20040>
fp16: False
loader: <torch.utils.data.dataloader.DataLoader object at 0x7f9002cd8d30>
mean: tensor([[[[123.6750]], [[116.2800]], [[103.5300]]]], device='cuda:0') //[torch.Size([1, 3, 1, 1])]
mixup_enabled: True
random_erasing: RandomErasing(p=0.25, mode=const, count=(1, 1))
sampler: <torch.utils.data.sampler.RandomSampler object at 0x7f8f79a7d040>
std: tensor([[[[58.3950]],[[57.1200]],[[57.3750]]]], device='cuda:0')//[torch.Size([1, 3, 1, 1])]

对spikformer V2的疑问

数据处理

从download->extract 的aedat格式是正常，本应该extract/xx/xx.aedat->events_up/xx/xx.npz，结果现在events_up/xx是空的文件夹，没有npz，尝试换台机器跑也是这个问题。

No module named 'sj_dropout'

Hi Dr. Zhou! A series of problems occurred when I run the "cifar10/train.py".

1. model cannot be imported into train.py, it shows a grey color
2. In model.py, there is an error "No module named 'sj_dropout' ". I have already searched on the internet but cannot find what the sj_dropout is.

Traceback (most recent call last): File "/home/PycharmProjects/spikformer/cifar10/train.py", line 42, in <module> import model File "/home/PycharmProjects/spikformer/cifar10/model.py", line 9, in <module> from sj_dropout import SN_Droppath ModuleNotFoundError: No module named 'sj_dropout'
Waiting for your reply! Thank you!

Energy consumption calculation on CIFAR10 dataset

Hi, thanks for you fancy work and publishing the code!

I reproduced you work with CIFAR10 dataset and calculated the energy consumption with this code (Spikingformer).

I got below outputs

SSA info:
{'depth': 4, 'Nheads': 12, 'embSize': 384, 'patchSize': 4, 'Tsteps': 4}
Firing rate of Q/K/V inputs in each block:
[[0.14379521053803118, 0.10392173096726212, 0.08082685798783845], [0.09622804452724094, 0.079531397717663, 0.07723161116053787], [0.0788185744534565, 0.07352738761449162, 0.0678107915799829], [0.0876925362250473, 0.08682637812593315, 0.06879291607986522]]
Number of operations: 0.005312266 G MACs, 0.2458512465306023 G ACs
Energy consumption: 0.24570254547754208 mJ
Nops: ['3703662346.0 Ops', '284837097.97817653 Ops', '1073133322.0 Ops']
Nparams: 9330874
Time cost: 0.4905501802762349 min

But, there are no accurate information about energy concumption on CIFAR10 in your paper, so I can't check if they are right values.

Can I make sure that is a properly reproduced results?

I used default config setting on cifar10.yaml i.e., Spikformer-4-384.

Thank you.

Pretrained Weight Access

Thanks for your wonderful work! But to further explore the power of Spikformer, the pretained weight might be necessary to move forward.

Could you please share the pretrained weight of your model? It is difficult for us to train such a huge model with a huge batchsize setting. Your pretrained model might be very helpful for futher work!

Confused about the use of layernormalization in your Imagenet training code.

Thanks for your kind sharing for your codes. You did a really good and meaningful research work in the field of SNN.

I just confused about # norm_layer=nn.LayerNorm # in your code of training Imagenet.

But in your paper, I found the state "We empircally find that the layer normalization (LN) does not apply to SNNs, so we use batch normalization (BN) instead." It seems a little bit inconsistent.

Many thanks!

Xin Wang

对spikformerv2的疑问

在第一篇论文里，spikeformerv1在imagenet数据集上最高精度不过74.81，在spikeformerv2中上升到了79.55，为什么？仅仅屏蔽了mixup和cutmix就可以做到这么大的提升吗

Fail to reproduce the Spikformer-4-256 on CIFAR-10

Hi dear authors,

Sorry for bothering you again. While training the Spikformer-4-256 model on CIFAR-10, I encountered a problem with the following error message: "AssertionError: dim 256 should be divided by num_heads 12."

I am seeking assistance to resolve this issue and was wondering if you could provide information regarding the value of num_heads used during your experiment. Any insights you can offer would be greatly appreciated.

Thank you very much for your time and assistance.

dvs128 code

Hello, ZK-Zhou,

I can not find the code about the dvs128Gesture, could you onload it?

Thank you very much!

TypeError: __init__() got an unexpected keyword argument 'pretrained_cfg'

I'm so sorry to bother you. I encountered the above problem when creating model. I want to know what may be the reason. Is it because of Timm's version?

Connecting Function on CIFAR10-DVS for SpikformerV2

Hello,
Since the code isn't uploaded yet, I was hoping you might be able to tell me what connecting function you used for spikformerv2 on CIFAR10-DVS. Was it ADD, IAND, or something else? Thanks!

how to get the pretrained weights of the models

i need well-trained weights of these models to do something else ？can you share them？

assert cupy is not None, 'CuPy is not installed!

ssh://[email protected]:23735/opt/conda/envs/py37/bin/python -u /home/u21b961009/jupyterlab/spikformer-main/imagenet/train.py
INFO:train:Training with a single process on 1 GPUs.
Training with a single process on 1 GPUs.
Traceback (most recent call last):
File "/home/u21b961009/jupyterlab/spikformer-main/imagenet/train.py", line 804, in
main()
File "/home/u21b961009/jupyterlab/spikformer-main/imagenet/train.py", line 353, in main
drop_block_rate=None,
File "/home/u21b961009/.local/lib/python3.7/site-packages/timm/models/factory.py", line 74, in create_model
model = create_fn(pretrained=pretrained, **kwargs)
File "/home/u21b961009/jupyterlab/spikformer-main/imagenet/model.py", line 254, in spikformer
**kwargs
File "/home/u21b961009/jupyterlab/spikformer-main/imagenet/model.py", line 195, in init
embed_dims=embed_dims)
File "/home/u21b961009/jupyterlab/spikformer-main/imagenet/model.py", line 125, in init
self.proj_lif = MultiStepLIFNode(tau=2.0, detach_reset=True, backend='cupy')
File "/home/u21b961009/.local/lib/python3.7/site-packages/spikingjelly/clock_driven/neuron.py", line 823, in init
check_backend(backend)
File "/home/u21b961009/.local/lib/python3.7/site-packages/spikingjelly/clock_driven/neuron.py", line 30, in check_backend
assert cupy is not None, 'CuPy is not installed! You can install it from "https://github.com/cupy/cupy".'
AssertionError: CuPy is not installed! You can install it from "https://github.com/cupy/cupy".

进程已结束,退出代码1

v2代码什么时候能开源呢

运行环境问题

Fail to Reproduce SpikFormer-8-512 on Imagenet

Hi dear authors,

Thank you guys for your amazing work. May I ask if there is anything important that requires modification based on the current code uploaded to github? I was trying to reproduce the result for Spikformer-8-512 on Imagenet. However, the result I got was quite different from the one shown in the paper.


When it comes to epoch 25, the top1-acc shown in the paper is clearly over 50%, while my reproduced result is barely over 40%. I used the exactly same code as in this github, except I used a batch size of 24 due to my GPU limitation.

Please enlight me. Thank you so much!

No checkpoint found at '/home/.../svit-8-512.pth.tar'

Hello, ZK-Zhou,

Could you tell me how to deal with the error " No checkpoint found at '/home/zhou/new_desk/Transformer-SNN/imagenet/src_5/pretrained/svit-8-512.pth.tar'"?

Thank you very much!

学习率初始化值问题

您好，您在论文中提到，在cifar10dvs与128dvs上：“The learning rate is initialized to 0.1”，但在train.py中lr使用的是0.001。我使用0.1的学习率对在cifar10dvs上训练，没有办法收敛；但用0.001的学习率训练时，复现结果与您论文相差不大。想问一下，您实际用的lr是多少呢？在论文中提到的lr为0.1是笔误吗？

关于spikformer应用于时间序列的想法

请问spikformer能应用于时序信息的提取处理嘛，譬如视频理解领域之类的
根据一二代的论文，似乎都主要应用于二维图像分类任务，代码中时序维度的T也是作为time step而不是时序

Problem about environment

In your requirement.txt timm==0.5.4 and torch==1.10.0+cu111 , but timm==0.5.4 requires torch>=1.4.0. And after upgrade the version of torch there is problem to train the models using 'python -m torch.distributed.launch --nproc_per_node=8 '.

torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1970, invalid usage (run with NCCL_DEBUG=WARN for details), NCCL version 2.20.5

The training code for CIFAR10-DVS dataset.

Hi Zhaokun. Thank you so much for providing the code for this work. Could you please share the training code for CIFAR10-DVS dataset? That would be very useful for my research. Thanks!!

Power calculation

Hi, thank you for your excellent work!

Could you please provide the code for power calculation?