{"id":20663763,"url":"https://github.com/vita-group/tost","last_synced_at":"2025-04-19T15:56:04.474Z","repository":{"id":45778404,"uuid":"508118038","full_name":"VITA-Group/ToST","owner":"VITA-Group","description":"[ICML2022] Training Your Sparse Neural Network Better with Any Mask. Ajay Jaiswal, Haoyu Ma, Tianlong Chen, ying Ding, and Zhangyang Wang","archived":false,"fork":false,"pushed_at":"2022-07-24T22:27:36.000Z","size":73,"stargazers_count":27,"open_issues_count":0,"forks_count":3,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-03-29T09:42:04.335Z","etag":null,"topics":["lottery-tickets","sparse-training","sparsity"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/VITA-Group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-06-28T01:44:42.000Z","updated_at":"2024-12-29T15:18:03.000Z","dependencies_parsed_at":"2022-08-19T18:20:35.958Z","dependency_job_id":null,"html_url":"https://github.com/VITA-Group/ToST","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FToST","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FToST/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FToST/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FToST/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/VITA-Group","download_url":"https://codeload.github.com/VITA-Group/ToST/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249731218,"owners_count":21317341,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["lottery-tickets","sparse-training","sparsity"],"created_at":"2024-11-16T19:19:44.586Z","updated_at":"2025-04-19T15:56:04.456Z","avatar_url":"https://github.com/VITA-Group.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Training Your Sparse Neural Network Better with Any Mask (ICML 2022)\n[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)\nPytorch Implementation of ICML 2022 \n\n#### [Code Under Development] \nWe will be releasing scripts to run our techniques efficiently for various architectures and dataset.\n\n## Installation\nWe recommend users to use `conda` to install the running environment. The following dependencies are required:\n```\nCUDA=11.1\nPython=3.7.7\npytorch=1.9.0\nsklearn=1.0.1\npillow=8.3.1\nopencv-python\nsvgpathtools\ncycler==0.10.0\nkiwisolver==1.1.0\nmatplotlib==3.1.1\nprotobuf==3.9.2\npyparsing==2.4.2\npython-dateutil==2.8.0\npytz==2019.2\nscipy==1.3.1\nseaborn==0.9.0\nsix==1.12.0\ntensorboardX==1.8\ntqdm==4.36.1\n```\nOur code should be compatible with pytorch\u003e=1.5.0\n\n\n## How to create the sparse mask for various SOTA pruning methods ? \n### Using pruning_techniques directory included with this repository :\n### Following is an example of creating a Lottery ticket mask :\n```\npython3 main.py --prune_type=lt --arch_type=resnet18 --dataset=cifar10 --prune_percent=10 --prune_iterations=5\n```\n- `--prune_type` : Type of pruning  \n- `--arch_type`\t : Type of architecture\n- `--dataset`\t: Choice of dataset \n- `--prune_percent`\t: Percentage of weight to be pruned after each cycle. \n- `--prune_iterations`\t: Number of cycle of pruning that should be done. \n- `--lr`\t: Learning rate \n- `--batch_size`\t: Batch size \n- `--end_iter`\t: Number of Epochs \n- `--gpu`\t: Decide Which GPU the program should use \n\n## Pruning methods:\n\n**RP:** random pruning\n\n**OMP:** oneshot pruning, magnitude pruning\n\n**GMP**: To prune, or not to prune: exploring the efficacy of pruning for model compression\n\n**TP:** Detecting Dead Weights and Units in Neural Networks, Page19 Table2.1 Taylor1Scorer (adding abs in our implementation) \n\n**SNIP:** SNIP: Single-shot network pruning based on connection sensitivity\n\n**GraSP:** Picking winning tickets before training by preserving gradient flow\n\n**SynFlow:** Pruning neural networks without any data by iteratively conserving synaptic flow\n\n## Code Details: \n\npruning methods implemented in **pruning_utils.py**\n\n**example.py** provides an simple examples\n\n## Training using soft-activation\n### Keep the mask identified using the previous pruning methods in the soft_activation/mask directory\n```\npython -u soft_activation/train_ticket.py --dataset cifar100 --activation swish  --arch resnet18  --manualSeed 42 --depth 18  --model [initial model path] --resume [resume_path] --save_dir [output_directory]  --gpu 3\n```\n### Activation based analysis\n```\npython activation_analysis.py --arch resnet18 --dataset cifar100 --manualSeed 42 --depth 18 --pretrained [pretrained checkpoint path]  --eval --gpu_id 1  --activation [relu/swish/mish] --layer [layer_number_to_analyse]\n```\n## Training using skip-connections\n### Keep the mask identified using the previous pruning methods in the skip_connection/mask directory\n```\npython skip_connection/train_ticket.py --dataset cifar100 --activation [activation_to_use]  --arch resnet18  --manualSeed 42 --depth 18  --model [initial model path] --resume [resume_path] --save_dir [output_directory]  --gpu 3  --gpu 0\n```\n\n## Training using label-smoothening\n### Keep the mask identified using the previous pruning methods in the label-smoothening/mask directory\n```\npython label-smoothening/train_ticket.py --dataset cifar100 --activation [activation_to_use]  --arch resnet18  --manualSeed 42 --depth 18  --model [initial model path] --resume [resume_path] --save_dir [output_directory]  --gpu 3  --gpu 0\n```\n\n## Training using LRsI\n### Keep the mask identified using the previous pruning methods in the LRsI/mask directory\n```\npython train_ticket.py --dataset cifar100 --activation relu  --arch resnet18  --manualSeed 42 --depth 18  --model [initial model path] --resume [resume_path] --save_dir [output_directory]  --gpu 2 --gradinit  --gradinit-alg sgd --gradinit-eta 0.1 --gradinit-gamma 1 --gradinit-normalize-grad --gradinit-lr 1e-2  --gradinit-min-scale 0.01 --gradinit-iters 180 --gradinit-grad-clip 1  \n```\n- `--gradinit` : Whether to use GradInit. \n- `--gradinit-alg`\t : The target optimization algorithm, deciding the direction of the first gradient step.\n- `--gradinit-eta`\t: The eta in GradInit.\n- `--gradinit-gamma`\t: The gradient norm constraint.\n- `--gradinit-normalize-grad`\t: Number of cycle of pruning that should be done. \n- `--gradinit-lr`\t: The learning rate of GradInit.\n- `--gradinit-min-scale`\t: The lower bound of the scaling factors\n- `--gradinit-iters`\t: Total number of iterations for GradInit.\n- `--gradinit-grad-clip`\t: Gradient clipping (per dimension) for GradInit\n\n**The code to support any architecture with only nn.Conv2d, nn.Linear and nn.BatchNorm2d as the parameterized layers. Simply call gradinit_utils.gradinit before your training loop.**\n\n\n## Acknowledgement\nThanks to Chen Zhu, Renkun Ni, Zheng Xu for opening source of their excellent implementation of GradInit works [GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training](https://github.com/zhuchen03/gradinit?utm_source=catalyzex.com).\n\n## Citation\n\nIf you find our code implementation helpful for your own resarch or work, please cite our paper.\n```\n@inproceedings{jaiswal2022ToST,\n  title={Training Your Sparse Neural Network Better with Any Mask},\n  author={Jaiswal, Ajay and Ma, Haoyu and Chen, Tianlong and Ding, Ying and Wang, Zhangyang},\n  booktitle={International Conference in Machine Learning},\n  year={2022}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvita-group%2Ftost","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvita-group%2Ftost","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvita-group%2Ftost/lists"}