{"id":15030479,"url":"https://github.com/eric-mingjie/rethinking-network-pruning","last_synced_at":"2025-05-16T06:04:14.987Z","repository":{"id":41337483,"uuid":"152529085","full_name":"Eric-mingjie/rethinking-network-pruning","owner":"Eric-mingjie","description":"Rethinking the Value of Network Pruning (Pytorch) (ICLR 2019)","archived":false,"fork":false,"pushed_at":"2020-06-07T10:04:45.000Z","size":168,"stargazers_count":1514,"open_issues_count":23,"forks_count":292,"subscribers_count":32,"default_branch":"master","last_synced_at":"2025-05-16T06:02:51.181Z","etag":null,"topics":["convolutional-neural-networks","deep-learning","network-pruning","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Eric-mingjie.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-10-11T04:03:00.000Z","updated_at":"2025-05-02T07:43:00.000Z","dependencies_parsed_at":"2022-09-01T12:01:08.478Z","dependency_job_id":null,"html_url":"https://github.com/Eric-mingjie/rethinking-network-pruning","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Eric-mingjie%2Frethinking-network-pruning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Eric-mingjie%2Frethinking-network-pruning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Eric-mingjie%2Frethinking-network-pruning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Eric-mingjie%2Frethinking-network-pruning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Eric-mingjie","download_url":"https://codeload.github.com/Eric-mingjie/rethinking-network-pruning/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254478186,"owners_count":22077675,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["convolutional-neural-networks","deep-learning","network-pruning","pytorch"],"created_at":"2024-09-24T20:13:27.731Z","updated_at":"2025-05-16T06:04:14.898Z","avatar_url":"https://github.com/Eric-mingjie.png","language":"Python","readme":"# Rethinking the Value of Network Pruning\nThis repository contains the code for reproducing the results, and trained ImageNet models, in the following paper:  \n\nRethinking the Value of Network Pruning. [[arXiv]](https://arxiv.org/abs/1810.05270) [[OpenReview]](https://openreview.net/forum?id=rJlnB3C5Ym)\n\n[Zhuang Liu](https://liuzhuang13.github.io/)\\*, [Mingjie Sun](https://eric-mingjie.github.io/)\\*, [Tinghui Zhou](https://people.eecs.berkeley.edu/~tinghuiz/), [Gao Huang](http://www.gaohuang.net/), [Trevor Darrell](https://people.eecs.berkeley.edu/~trevor/) (\\* equal contribution).\n\nICLR 2019. Also [Best Paper Award](https://nips.cc/Conferences/2018/Schedule?showEvent=10941) at NIPS 2018 Workshop on Compact Deep Neural Networks.\n\nSeveral pruning methods' implementations contained in this repo can also be readily used for other research purposes.\n\n## Paper Summary\n\n\u003cdiv align=center\u003e\n\u003cimg src=\"https://user-images.githubusercontent.com/8370623/67258108-e98f9c80-f443-11e9-9146-9a8333c6f318.png\" width=\"360\"\u003e\n\u003c/div\u003e\n\n\u003cdiv align=center\u003e\nFig 1: A typical three-stage network pruning\npipeline.\n\u003c/div\u003e\n\n\u003cbr\u003e\n\nOur paper shows that for **structured** pruning, **training the pruned model from scratch can almost always achieve comparable or higher level of accuracy than the model obtained from the typical \"training, pruning and fine-tuning\" (Fig. 1) procedure**. We conclude that for those pruning methods:\n\n1. Training a large, over-parameterized model is often not necessary to obtain an efficient final model.\n2. Learned “important” weights of the large model are typically not useful for the small pruned model. \n3. The pruned architecture itself, rather than a set of inherited “important” weights, is more crucial to the efficiency in the final model, which suggests that in some cases pruning can be useful as an architecture search paradigm. \n\nOur results suggest the need for more careful baseline evaluations in future research on structured pruning methods. \n\n\u003cdiv align=center\u003e\n\u003cimg src=\"https://user-images.githubusercontent.com/8370623/67257959-3b83f280-f443-11e9-8e4a-3afec30cff45.png\" width=\"400\"\u003e\n\u003c/div\u003e\n\nFig 2: Difference between predefined and automatically discovered target architectures, in channel pruning. The pruning ratio x is user-specified, while a, b, c, d are determined by the pruning algorithm. Unstructured sparse pruning can also be viewed as automatic. Our finding has different implications for predefined and automatic methods: for a predefined method, it is possible to skip the traditional \"training, pruning and fine-tuning\" pipeline and directly train the pruned model; for automatic methods, the pruning can be seen as a form of architecture learning.\n\n\u003cbr\u003e\n\nWe also compare with the \"[Lottery Ticket Hypothesis](https://arxiv.org/abs/1803.03635)\" (Frankle \u0026 Carbin 2019), and find that with optimal learning rate, the \"winning ticket\" initialization as used in Frankle \u0026 Carbin (2019) does not bring improvement over random initialization. For more details please refer to our paper.\n\n## Implementation\nWe evaluated the following seven pruning methods. \n\n1. [L1-norm based channel pruning](https://arxiv.org/abs/1608.08710)\n2. [ThiNet](https://arxiv.org/abs/1707.06342)\n3. [Regression based feature reconstruction](https://arxiv.org/abs/1707.06168)\n4. [Network Slimming](https://arxiv.org/abs/1708.06519)\n5. [Sparse Structure Selection](https://arxiv.org/abs/1707.01213)\n6. [Soft filter pruning](https://www.ijcai.org/proceedings/2018/0309.pdf)\n7. [Unstructured weight-level pruning](https://arxiv.org/abs/1506.02626)\n\nThe first six is structured while the last one is unstructured (or sparse). For CIFAR, our code is based on [pytorch-classification](https://github.com/bearpaw/pytorch-classification) and [network-slimming](https://github.com/Eric-mingjie/network-slimming). For ImageNet, we use the [official Pytorch ImageNet training code](https://github.com/pytorch/examples/blob/0.3.1/imagenet/main.py). The instructions and models are in each subfolder.\n\nFor experiments on [The Lottery Ticket Hypothesis](https://arxiv.org/abs/1803.03635), please refer to the folder [cifar/lottery-ticket](https://github.com/Eric-mingjie/rethinking-network-pruning/tree/master/cifar/lottery-ticket).\n\nOur experiment environment is Python 3.6 \u0026 PyTorch 0.3.1. \n\n## Contact\nFeel free to discuss papers/code with us through issues/emails!\n\nsunmj15 at gmail.com  \nliuzhuangthu at gmail.com\n\n## Citation\nIf you use our code in your research, please cite:\n```\n@inproceedings{liu2018rethinking,\n  title={Rethinking the Value of Network Pruning},\n  author={Liu, Zhuang and Sun, Mingjie and Zhou, Tinghui and Huang, Gao and Darrell, Trevor},\n  booktitle={ICLR},\n  year={2019}\n}\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feric-mingjie%2Frethinking-network-pruning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feric-mingjie%2Frethinking-network-pruning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feric-mingjie%2Frethinking-network-pruning/lists"}