{"id":13644074,"url":"https://github.com/BICLab/Spike-Driven-Transformer","last_synced_at":"2025-04-21T06:32:56.510Z","repository":{"id":178673699,"uuid":"662105186","full_name":"BICLab/Spike-Driven-Transformer","owner":"BICLab","description":"Offical implementation of \"Spike-driven Transformer\" (NeurIPS2023)","archived":false,"fork":false,"pushed_at":"2024-03-18T06:56:13.000Z","size":2829,"stargazers_count":219,"open_issues_count":0,"forks_count":17,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-11-09T16:43:45.664Z","etag":null,"topics":["neuromorphic-computing","spike-driven","spiking-neural-network","spiking-transformer","transformer-based-snn"],"latest_commit_sha":null,"homepage":"https://openreview.net/forum?id=9FmolyOHi5","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BICLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-07-04T11:14:08.000Z","updated_at":"2024-11-08T17:44:13.000Z","dependencies_parsed_at":null,"dependency_job_id":"c9e12c12-0f64-441b-acec-d8159a1469ff","html_url":"https://github.com/BICLab/Spike-Driven-Transformer","commit_stats":null,"previous_names":["biclab/spike-driven-transformer"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BICLab%2FSpike-Driven-Transformer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BICLab%2FSpike-Driven-Transformer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BICLab%2FSpike-Driven-Transformer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BICLab%2FSpike-Driven-Transformer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BICLab","download_url":"https://codeload.github.com/BICLab/Spike-Driven-Transformer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250008284,"owners_count":21359959,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["neuromorphic-computing","spike-driven","spiking-neural-network","spiking-transformer","transformer-based-snn"],"created_at":"2024-08-02T01:01:57.176Z","updated_at":"2025-04-21T06:32:51.496Z","avatar_url":"https://github.com/BICLab.png","language":"Python","funding_links":[],"categories":["Object Detection Applications","Applications"],"sub_categories":[],"readme":"# Spike-Driven Transformer ([NeurIPS2023](https://openreview.net/forum?id=9FmolyOHi5))\n\n[Man Yao](https://scholar.google.com/citations?user=eE4vvp0AAAAJ), [Jiakui Hu](https://github.com/jkhu29), [Zhaokun Zhou](https://github.com/ZK-Zhou), [Li Yuan](https://yuanli2333.github.io/), [Yonghong Tian](https://scholar.google.com/citations?user=fn6hJx0AAAAJ), [Bo Xu](), [Guoqi Li](https://scholar.google.com/citations?user=qCfE--MAAAAJ\u0026)\n\nBICLab, Institute of Automation, Chinese Academy of Sciences\n\n---\n\n:rocket:  :rocket:  :rocket: **News**:\n\n- **Jul. 04, 2023**: Release the code for training and testing.\n- **Sep. 22, 2023**: Accepted as poster in NeurIPS2023.\n- **Sep. 30, 2023**: Release the configs and pre-trained parameters on IN1K.\n- **Feb. 15. 2024**: The [Spike-Driven Transformer V2](https://github.com/BICLab/Spike-Driven-Transformer-V2), which achieves 80.0% acc on IN1K, is now available.\n\n## Abstract\n\nSpiking Neural Networks (SNNs) provide an energy-efficient deep learning option due to their unique spike-based event-driven (i.e., spike-driven) paradigm. In this paper, we incorporate the spike-driven paradigm into Transformer by the proposed Spike-driven Transformer with four unique properties: i) **Event-driven**, no calculation is triggered when the input of Transformer is zero; ii) **Binary spike communication**, all matrix multiplications associated with the spike matrix can be transformed into sparse additions; iii) **Self-attention with linear complexity at both token and channel dimensions**; iv) The operations between spike-form Query, Key, and Value are mask and addition. Together, **there are only sparse addition operations** in the Spike-driven Transformer. To this end, we design a novel Spike-Driven Self-Attention (SDSA), which exploits only mask and addition operations without any multiplication, and thus having up to **87.2× lower** computation energy than vanilla self-attention. Especially in SDSA, the matrix multiplication between Query, Key, and Value is designed as the mask operation. In addition, we rearrange all residual connections in the vanilla Transformer before the activation functions to ensure that all neurons transmit binary spike signals. It is shown that the Spike-driven Transformer can achieve **77.1% top-1** accuracy on ImageNet-1K, which is the state-of-the-art result in the SNN field.\n\n![SDSA](./imgs/Fig_1_main_idea.png)\n\n## Requirements\n\n```python3\ntimm == 0.6.12\n1.10.0 \u003c= pytorch \u003c 2.0.0\ncupy\nspikingjelly == 0.0.0.0.12\ntensorboard\n```\n\n\u003e !!! Please install the spikingjelly and tensorboard correctly before raising issues about requirements. !!!\n\n## Results on Imagenet-1K\n\n|        **model**         | **T** | **layers** | **channels** | **Top-1 Acc** | **Power(mj)** | **Models** |\n| :----------------------: | :---: | :--------: | :----------: | :-----------: | :-----------: | :--------: |\n| Spike-Driven Transformer |   4   |     8      |     384      |   **72.28**   |   **3.90**    |    [link](https://drive.google.com/file/d/10oH_zkwB4FDtFLgmZ_lI8e0tFjRzrXyD/view?usp=sharing)    |\n| Spike-Driven Transformer |   4   |     6      |     512      |   **74.11**   |   **3.56**    |    [link](https://drive.google.com/file/d/1hsShpFBKYpMK2TmpuoyBFORcLAuMHrx7/view?usp=sharing)    |\n| Spike-Driven Transformer |   4   |     8      |     512      |   **74.57**   |   **4.50**    |    [link](https://drive.google.com/file/d/1n59WNSBgP2VyAW2nfJX2Wvx5rgNJEMXI/view?usp=sharing)    |\n| Spike-Driven Transformer |   4   |     10     |     512      |   **74.66**   |   **5.53**    |    [link](https://drive.google.com/file/d/1l-c3QY5r4IFmYUmGPZXRHP_W7iC1sdP8/view?usp=sharing)    |\n| Spike-Driven Transformer |   4   |     8      |     768      |   **77.07**   |   **6.09**    |    [link](https://drive.google.com/file/d/1R-MaeFV8d2Y0pIGBSjklOGWhaF8dLHf4/view?usp=sharing)    |\n\n## Train \u0026 Test\n\n![The architecture of Spike-Driven-Transformer.](./imgs/Fig_2_network_architecture.png)\n\nThe hyper-parameters are in `./conf/`.\n\n\nTrain:\n\n```shell\nCUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --nproc_per_node=1 --master_port 29501 train.py -c /the/path/of/conf --model sdt --spike-mode lif\n```\n\nTest:\n\n```shell\nCUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --nproc_per_node=1 --master_port 29501 firing_num.py -c /the/path/of/conf --model sdt --spike-mode lif --resume /the/path/of/parameters --no-resume-opt\n\n# for 288 x 288 resolution\nCUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --nproc_per_node=1 --master_port 29501 firing_num.py -c /the/path/of/conf --model sdt --spike-mode lif --resume /the/path/of/parameters --no-resume-opt --large-valid\n```\n\nResult and explainability:\n\n![The Attention Map of Spike-Driven Transformer in ImageNet.](./imgs/Fig_3_attention_map.png)\n\n## Data Prepare\n\n- use `PyTorch` to load the CIFAR10 and CIFAR100 dataset.\n- use `SpikingJelly` to prepare and load the Gesture and CIFAR10-DVS dataset.\n\nTree in `./data/`.\n\n```shell\n.\n├── cifar-100-python\n├── cifar-10-batches-py\n├── cifar10-dvs\n│   ├── download\n│   ├── events_np\n│   ├── extract\n│   ├── frames_number_10_split_by_number\n│   └── frames_number_16_split_by_number\n├── cifar10-dvs-tet\n│   ├── test\n│   └── train\n└── DVSGesturedataset\n    ├── download\n    ├── events_np\n    │   ├── test\n    │   └── train\n    ├── extract\n    │   └── DvsGesture\n    ├── frames_number_10_split_by_number\n    │   ├── download\n    │   ├── test\n    │   └── train\n    └── frames_number_16_split_by_number\n        ├── test\n        └── train\n```\n\nImageNet with the following folder structure, you can extract imagenet by this [script](https://gist.github.com/BIGBALLON/8a71d225eff18d88e469e6ea9b39cef4).\n\n```shell\n│imagenet/\n├──train/\n│  ├── n01440764\n│  │   ├── n01440764_10026.JPEG\n│  │   ├── n01440764_10027.JPEG\n│  │   ├── ......\n│  ├── ......\n├──val/\n│  ├── n01440764\n│  │   ├── ILSVRC2012_val_00000293.JPEG\n│  │   ├── ILSVRC2012_val_00002138.JPEG\n│  │   ├── ......\n│  ├── ......\n```\n\n## Contact Information\n\n```\n@inproceedings{yao2023spikedriven,\ntitle={Spike-driven Transformer},\nauthor={Man Yao and JiaKui Hu and Zhaokun Zhou and Li Yuan and Yonghong Tian and Bo XU and Guoqi Li},\nbooktitle={Thirty-seventh Conference on Neural Information Processing Systems},\nyear={2023},\nurl={https://openreview.net/forum?id=9FmolyOHi5}\n}\n```\n\nFor help or issues using this git, please submit a GitHub issue.\n\nFor other communications related to this git, please contact `manyao@ia.ac.cn` and `jkhu29@stu.pku.edu.cn`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FBICLab%2FSpike-Driven-Transformer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FBICLab%2FSpike-Driven-Transformer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FBICLab%2FSpike-Driven-Transformer/lists"}