{"id":13644078,"url":"https://github.com/BICLab/Spike-Driven-Transformer-V2","last_synced_at":"2025-04-21T06:32:56.202Z","repository":{"id":222615484,"uuid":"754146150","full_name":"BICLab/Spike-Driven-Transformer-V2","owner":"BICLab","description":"Offical implementation of \"Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips\" (ICLR2024)","archived":false,"fork":false,"pushed_at":"2024-05-10T15:06:37.000Z","size":5092,"stargazers_count":136,"open_issues_count":1,"forks_count":18,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-11-09T16:43:44.431Z","etag":null,"topics":["neuromorphic-computing","neuromorphic-hardware","spiking-neural-networks"],"latest_commit_sha":null,"homepage":"https://openreview.net/forum?id=1SIBN5Xyw7","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BICLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-07T13:45:33.000Z","updated_at":"2024-11-06T06:47:28.000Z","dependencies_parsed_at":"2024-04-17T06:22:29.871Z","dependency_job_id":"de585e4c-cd1e-49a5-ba07-5ae95bd207e6","html_url":"https://github.com/BICLab/Spike-Driven-Transformer-V2","commit_stats":null,"previous_names":["biclab/spike-driven-transformer-v2"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BICLab%2FSpike-Driven-Transformer-V2","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BICLab%2FSpike-Driven-Transformer-V2/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BICLab%2FSpike-Driven-Transformer-V2/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BICLab%2FSpike-Driven-Transformer-V2/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BICLab","download_url":"https://codeload.github.com/BICLab/Spike-Driven-Transformer-V2/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250008284,"owners_count":21359959,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["neuromorphic-computing","neuromorphic-hardware","spiking-neural-networks"],"created_at":"2024-08-02T01:01:57.226Z","updated_at":"2025-04-21T06:32:51.390Z","avatar_url":"https://github.com/BICLab.png","language":"Python","funding_links":[],"categories":["Object Detection Applications","Applications"],"sub_categories":[],"readme":"# Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips ([ICLR2024](https://openreview.net/forum?id=1SIBN5Xyw7))\n\n[Man Yao](https://scholar.google.com/citations?user=eE4vvp0AAAAJ), [Jiakui Hu](https://github.com/jkhu29), [Tianxiang Hu](), [Yifan Xu](https://scholar.google.com/citations?hl=zh-CN\u0026user=pbcoTgsAAAAJ), [Zhaokun Zhou](https://scholar.google.com/citations?user=4nz-h1QAAAAJ), [Yonghong Tian](https://scholar.google.com/citations?user=fn6hJx0AAAAJ), [Bo Xu](), [Guoqi Li](https://scholar.google.com/citations?user=qCfE--MAAAAJ\u0026)\n\nBICLab, Institute of Automation, Chinese Academy of Sciences\n\n---\n\n:rocket:  :rocket:  :rocket: **News**:\n\n- **Jan. 16, 2024**: Accepted as poster in ICLR2024.\n- **Feb. 15, 2024**: Release the training and inference codes in classification tasks.\n- **Apr. 19, 2024**: Release the [pre-trained ckpts and training logs](https://drive.google.com/drive/folders/12JcIRG8BF6JcgPsXIetSS14udtHXeSSx?usp=sharing) of SDT-v2.\n\nTODO:\n\n- [x] Upload train and test scripts.\n- [x] Upload checkpoints.\n\n## Abstract\n\nNeuromorphic computing, which exploits Spiking Neural Networks (SNNs) on neuromorphic chips, is a promising energy-efficient alternative to traditional AI. CNN-based SNNs are the current mainstream of neuromorphic computing. By contrast, no neuromorphic chips are designed especially for Transformer-based SNNs, which have just emerged, and their performance is only on par with CNN-based SNNs, offering no distinct advantage. In this work, we propose a general Transformer-based SNN architecture, termed as \"Meta-SpikeFormer\", whose goals are: (1) **Lower-power**, supports the spike-driven paradigm that there is only sparse addition in the network; (2) **Versatility**, handles various vision tasks; (3) **High-performance**, shows overwhelming performance advantages over CNN-based SNNs; (4) **Meta-architecture**, provides inspiration for future next-generation Transformer-based neuromorphic chip designs. Specifically, we extend the [Spike-driven Transformer](https://github.com/BICLab/Spike-Driven-Transformer) into a meta architecture, and explore the impact of structure, spike-driven self-attention, and skip connection on its performance. On ImageNet-1K, Meta-SpikeFormer achieves **80.0% top-1 accuracy** (55M), surpassing the current state-of-the-art (SOTA) SNN baselines (66M) by 3.7%. This is the first direct training SNN backbone that can simultaneously **supports classification, detection, and segmentation**, obtaining SOTA results in SNNs. Finally, we discuss the inspiration of the meta SNN architecture for neuromorphic chip design.\n\n![V2](./img/300_spike_driven_transformer_v2_me.png)\n\n## Classification\n\n### Requirements\n\n```python3\npytorch \u003e= 2.0.0\ncupy\nspikingjelly == 0.0.0.0.12\n```\n\n### Results on Imagenet-1K\n\nPre-trained ckpts and training logs of 55M: [here](https://drive.google.com/drive/folders/12JcIRG8BF6JcgPsXIetSS14udtHXeSSx?usp=sharing).\n\n### Train \u0026 Test\n\nThe hyper-parameters are in `./conf/`.\n\nTrain:\n\n```shell\ntorchrun --standalone --nproc_per_node=8 \\\n  main_finetune.py \\\n  --batch_size 128 \\\n  --blr 6e-4 \\\n  --warmup_epochs 10 \\\n  --epochs 200 \\\n  --model metaspikformer_8_512 \\\n  --data_path /your/data/path \\\n  --output_dir outputs/T1 \\\n  --log_dir outputs/T1 \\\n  --model_mode ms \\\n  --dist_eval\n```\n\nFinetune:\n\n\u003e Please download caformer_b36_in21_ft1k.pth first following [PoolFormer](https://github.com/sail-sg/poolformer).\n\n```shell\ntorchrun --standalone --nproc_per_node=8 \\\n  main_finetune.py \\\n  --batch_size 24 \\\n  --blr 2e-5 \\\n  --warmup_epochs 5 \\\n  --epochs 50 \\\n  --model metaspikformer_8_512 \\\n  --data_path /your/data/path \\\n  --output_dir outputs/T4 \\\n  --log_dir outputs/T4 \\\n  --model_mode ms \\\n  --dist_eval \\\n  --finetune /your/ckpt/path \\\n  --time_steps 4 \\\n  --kd \\\n  --teacher_model caformer_b36_in21ft1k \\\n  --distillation_type hard\n```\n\nTest:\n\n```shell\npython main_finetune.py --batch_size 128 --model metaspikformer_8_512 --data_path /your/data/path --eval --resume /your/ckpt/path\n```\n\n### Data Prepare\n\nImageNet with the following folder structure, you can extract imagenet by this [script](https://gist.github.com/BIGBALLON/8a71d225eff18d88e469e6ea9b39cef4).\n\n```shell\n│imagenet/\n├──train/\n│  ├── n01440764\n│  │   ├── n01440764_10026.JPEG\n│  │   ├── n01440764_10027.JPEG\n│  │   ├── ......\n│  ├── ......\n├──val/\n│  ├── n01440764\n│  │   ├── ILSVRC2012_val_00000293.JPEG\n│  │   ├── ILSVRC2012_val_00002138.JPEG\n│  │   ├── ......\n│  ├── ......\n```\n\n## Contact Information\n\n```\n@inproceedings{\nyao2024spikedriven,\ntitle={Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips},\nauthor={Man Yao and JiaKui Hu and Tianxiang Hu and Yifan Xu and Zhaokun Zhou and Yonghong Tian and Bo XU and Guoqi Li},\nbooktitle={The Twelfth International Conference on Learning Representations},\nyear={2024},\nurl={https://openreview.net/forum?id=1SIBN5Xyw7}\n}\n```\n\nFor help or issues using this git, please submit a GitHub issue.\n\nFor other communications related to this git, please contact `manyao@ia.ac.cn` and `jkhu29@stu.pku.edu.cn`.\n\n## Thanks\n\nOur implementation is mainly based on the following codebases. We gratefully thank the authors for their wonderful works.\n\n[deit](https://github.com/facebookresearch/deit)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FBICLab%2FSpike-Driven-Transformer-V2","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FBICLab%2FSpike-Driven-Transformer-V2","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FBICLab%2FSpike-Driven-Transformer-V2/lists"}