{"id":20663683,"url":"https://github.com/vita-group/svite","last_synced_at":"2025-04-19T15:55:50.490Z","repository":{"id":43035719,"uuid":"371832980","full_name":"VITA-Group/SViTE","owner":"VITA-Group","description":"[NeurIPS'21] \"Chasing Sparsity in Vision Transformers: An End-to-End Exploration\" by Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, Zhangyang Wang","archived":false,"fork":false,"pushed_at":"2023-12-01T06:05:30.000Z","size":630,"stargazers_count":89,"open_issues_count":3,"forks_count":12,"subscribers_count":14,"default_branch":"main","last_synced_at":"2025-03-29T09:41:59.531Z","etag":null,"topics":["dynamic-sparsity","efficient-transformers","model-compression","pruning","sparse-training","token-slimming","vision-transformers"],"latest_commit_sha":null,"homepage":"https://tianlong-chen.github.io/about/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/VITA-Group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-05-28T22:39:32.000Z","updated_at":"2025-02-08T09:57:33.000Z","dependencies_parsed_at":"2024-11-16T19:29:57.532Z","dependency_job_id":null,"html_url":"https://github.com/VITA-Group/SViTE","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FSViTE","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FSViTE/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FSViTE/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FSViTE/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/VITA-Group","download_url":"https://codeload.github.com/VITA-Group/SViTE/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249731218,"owners_count":21317341,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dynamic-sparsity","efficient-transformers","model-compression","pruning","sparse-training","token-slimming","vision-transformers"],"created_at":"2024-11-16T19:19:17.057Z","updated_at":"2025-04-19T15:55:50.477Z","avatar_url":"https://github.com/VITA-Group.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Chasing Sparsity in Vision Transformers: An End-to-End Exploration\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)\n\nCodes for [NeurIPS'21] [Chasing Sparsity in Vision Transformers: An End-to-End Exploration](https://arxiv.org/pdf/2106.04533.pdf).\n\nTianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, Zhangyang Wang\n\n\n\n## Overall Results\n\n\u003cimg src = \"./Figs/res.png\" width=\"60%\" hight=\"60%\"\u003e\n\nExtensive results on ImageNet with diverse ViT backbones validate the effectiveness of our proposals which obtain significantly reduced computational cost and almost unimpaired generalization. Perhaps most surprisingly, we find that the proposed sparse (co-)training can even *improve the ViT accuracy* rather than compromising it, making sparsity a tantalizing “free lunch”. For example, our sparsified DeiT-Small at (5%, 50%) sparsity for (data, architecture), improves 0.28% top-1 accuracy, and meanwhile enjoys 49.32% FLOPs and 4.40% running time savings.\n\n\n\n## Proposed Framework of SViTE\n\n![](./Figs/framework.png)\n\n\n\n## Implementations of SViTE\n\n### Set Environment\n\n```shell\nconda create -n vit python=3.6\n\npip install torch==1.7.1+cu101 torchvision==0.8.2+cu101 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html\n\npip install tqdm scipy timm\n\ngit clone https://github.com/NVIDIA/apex\n\ncd apex\n\npip install -v --disable-pip-version-check --no-cache-dir --global-option=\"--cpp_ext\" --global-option=\"--cuda_ext\" ./\n\npip install -v --disable-pip-version-check --no-cache-dir ./\n```\n\n### Cmd\n\nCommand for unstructured sparsity, i.e., SViTE.\n\n- SViTE-Small\n\n```shell\nbash cmd/ vm/0426/vm1.sh 0,1,2,3,4,5,6,7\n```\n\nDetails\n\n```shell\nCUDA_VISIBLE_DEVICES=$1 \\\npython -m torch.distributed.launch \\\n    --nproc_per_node=8 \\\n    --use_env main.py \\\n    --model deit_small_patch16_224 \\\n    --epochs 600 \\\n    --batch-size 64 \\\n    --data-path ../../imagenet \\\n    --output_dir ./small_dst_uns_0426_vm1 \\\n    --dist_url tcp://127.0.0.1:23305 \\\n    --sparse_init fixed_ERK \\\n    --density 0.4 \\\n    --update_frequency 15000 \\\n    --growth gradient \\\n    --death magnitude \\\n    --redistribution none\n```\n\n- SViTE-Base\n\n```shell\nbash cmd/ vm/0426/vm3.sh 0,1,2,3,4,5,6,7\n```\n\nDetails\n\n```shell\nCUDA_VISIBLE_DEVICES=$1 \\\npython -m torch.distributed.launch \\\n    --nproc_per_node=8 \\\n    --use_env main.py \\\n    --model deit_base_patch16_224 \\\n    --epochs 600 \\\n    --batch-size 128 \\\n    --data-path ../../imagenet \\\n    --output_dir ./base_dst_uns_0426_vm3 \\\n    --dist_url tcp://127.0.0.1:23305 \\\n    --sparse_init fixed_ERK \\\n    --density 0.4 \\\n    --update_frequency 7000 \\\n    --growth gradient \\\n    --death magnitude \\\n    --redistribution none\n```\n\n**Remark.** More commands can be found under the \"cmd\" folder.\n\n## Citation\n\n```\n@misc{chen2021chasing,\n      title={Chasing Sparsity in Vision Transformers:An End-to-End Exploration}, \n      author={Tianlong Chen and Yu Cheng and Zhe Gan and Lu Yuan and Lei Zhang and Zhangyang Wang},\n      year={2021},\n      eprint={2106.04533},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV}\n}\n```\n\n\n\n## Acknowledge Related Repos\n\nViT : https://github.com/jeonsworld/ViT-pytorch\n\nViT : https://github.com/google-research/vision_transformer\n\nRig : https://github.com/google-research/rigl\n\nDeiT: https://github.com/facebookresearch/deit\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvita-group%2Fsvite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvita-group%2Fsvite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvita-group%2Fsvite/lists"}