{"id":13546738,"url":"https://github.com/huggingface/nn_pruning","last_synced_at":"2025-04-02T18:32:28.162Z","repository":{"id":37452204,"uuid":"323640052","full_name":"huggingface/nn_pruning","owner":"huggingface","description":"Prune a model while finetuning or training.","archived":false,"fork":false,"pushed_at":"2022-06-21T12:53:51.000Z","size":31595,"stargazers_count":401,"open_issues_count":20,"forks_count":60,"subscribers_count":45,"default_branch":"main","last_synced_at":"2025-03-25T07:38:45.716Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/huggingface.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-12-22T13:53:42.000Z","updated_at":"2025-03-24T13:09:29.000Z","dependencies_parsed_at":"2022-07-12T22:30:31.162Z","dependency_job_id":null,"html_url":"https://github.com/huggingface/nn_pruning","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fnn_pruning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fnn_pruning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fnn_pruning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fnn_pruning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/huggingface","download_url":"https://codeload.github.com/huggingface/nn_pruning/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246870144,"owners_count":20847251,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T12:00:44.087Z","updated_at":"2025-04-02T18:32:26.554Z","avatar_url":"https://github.com/huggingface.png","language":"Jupyter Notebook","funding_links":[],"categories":["🤗 Official Libraries","Jupyter Notebook"],"sub_categories":[],"readme":"# Neural Networks Block Movement Pruning\n\n\nAn interactive version of this site is available [here](https://huggingface.github.io/nn_pruning/).\n\n\n\n***[Movement](https://arxiv.org/abs/2005.07683) [pruning](https://github.com/huggingface/transformers/tree/master/examples/research_projects/movement-pruning)** has been proved as a **very efficient\nmethod to prune networks in a unstructured manner**. High levels of sparsity can be reached with a minimal of accuracy loss.\nThe resulting sparse networks can be **compressed heavily**,\nsaving a lot of permanent storage space on servers or devices, and bandwidth, an important advantage for edge devices.\n**But efficient inference with unstructured sparsity is hard.**\nSome degree of structure is necessary to use the intrinsic parallel nature of today hardware.\n**Block Movement Pruning** work extends the original method and explore **semi-structured and structured variants** of Movement Pruning.\nYou can read more about block sparsity and why it matters for performance on these [blog](https://medium.com/huggingface/is-the-future-of-neural-networks-sparse-an-introduction-1-n-d03923ecbd70) [posts](https://medium.com/huggingface/sparse-neural-networks-2-n-gpu-performance-b8bc9ce950fc).*\n\n## Documentation\nThe documentation is [here](docs/HOWTO.md).\n\n## Installation\n\n### User installlation\n\nYou can install `nn_pruning` using `pip` as follows:\n\n```\npython -m pip install -U nn_pruning\n```\n\n### Developer installation\n\nTo install the latest state of the source code, first clone the repository\n\n```\ngit clone https://github.com/huggingface/nn_pruning.git\n```\n\nand then install the required dependencies:\n\n```\ncd nn_pruning\npython -m pip install -e \".[dev]\"\n```\n\nAfter the installation is completed, you can launch the test suite from the root of the repository\n\n```\npytest nn_pruning\n```\n\n## Results\n\n### Squad V1\nThe experiments were done first on SQuAD v1.\n\nTwo networks were tested: BERT-base, and BERT-large.\n\nVery significant speedups were obtained with limited drop in accuracy.\n\nHere is a selection of the networks that are obtained through the different variant method variants.\n\nThe original \"large\" and \"base\" finedtuned models were added in the table for comparison.\n\nThe \"BERT version\" column shows which base network was pruned.\nThe parameter count column is relative to linear layers, which contain most of the model parameters (with the embeddings being most of the remaining parameters).\n\n**F1 difference, speedups and parameters counts are all relative to BERT-base to ease practical comparison.**\n\n|                                              Model                                               |  Type   |   method    | Params  |   F1    | F1 diff | Speedup |\n|--------------------------------------------------------------------------------------------------|---------|-------------|---------|---------|---------|---------|\n|**[#1](https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad)**            |**large**|**-**        |**+166%**|**93.15**|**+4.65**|**0.35x**|\n|#2                                                                                                |large    |hybrid-filled|-17%     |    91.03|+2.53    |0.92x    |\n|#3                                                                                                |large    |hybrid-filled|-40%     |    90.16|+1.66    |1.03x    |\n|#4                                                                                                |base     |hybrid-filled|-59%     |    88.72|+0.22    |1.84x    |\n|**[#5](https://huggingface.co/csarron/bert-base-uncased-squad-v1)**                               |**base** |**-**        |**+0%**  |**88.5** |**+0.00**|**1.00x**|\n|#6                                                                                                |base     |hybrid-filled|-65%     |    88.25|-0.25    |1.98x    |\n|**[#7](https://huggingface.co/madlag/bert-base-uncased-squadv1-x2.44-f87.7-d26-hybrid-filled-v1)**|base     |hybrid-filled|-74%     |    87.71|-0.79    |2.44x    |\n|#8                                                                                                |base     |hybrid-filled|-73%     |    87.23|-1.27    |2.60x    |\n|#9                                                                                                |base     |hybrid-filled|-74%     |    86.69|-1.81    |2.80x    |\n|#10                                                                                               |base     |struct       |-86%     |    85.52|-2.98    |3.64x    |\n\n\n\n\n### Main takeaways\n- network #2: pruned from BERT-large, it's significantly more accurate than BERT-base, but have a similar size and speed.\n- network #3: pruned from BERT-large, it is finally 40% smaller but significantly better than a BERT-base, and still as fast.\n\nThat means that starting from a larger networks is beneficial on all metrics, even absolute size, something observed in the [Train Large, Then Compress](https://arxiv.org/abs/2002.11794) paper.\n\n- network #4: we can shrink BERT-base by ~60%, speedup inference by 1.8x and still have a ***better*** network\n- networks #N: we can select a **tradeoff between speed and accuracy**, depending on the final application.\n- last network: pruned using a slightly different \"structured pruning\" method that gives faster networks but with a significant drop in F1.\n\n**Additional remarks**\n- The parameter reduction of the BERT-large networks are actually higher compared to the original network: 40% smaller than BERT-base means actually 77% smaller than BERT-large.\nWe kept here the comparison with BERT-base numbers as it's what matters on a practical point of view.\n- The \"theoretical speedup\" is a speedup of linear layers (actual number of flops), something that seems to be equivalent to the measured speedup in some papers.\nThe speedup here is measured on a 3090 RTX, using the HuggingFace transformers library, using Pytorch cuda timing features, and so is 100% in line with real-world speedup.\n\n### Example \"Hybrid filled\" Network\n\nHere are some visualizations of the pruned network [#7](https://huggingface.co/madlag/bert-base-uncased-squadv1-x2.44-f87.7-d26-hybrid-filled-v1).\nIt is using the \"Hybrid filled\" method:\n- Hybrid : prune using blocks for attention and rows/columns for the two large FFNs.\n- Filled : remove empty heads and empty rows/columns of the FFNs, then re-finetune the previous network, letting the zeros in non-empty attention heads evolve and so regain some accuracy while keeping the same network speed.\n\nYou can see that the results linear layers are all actually \"dense\" (hover on the graph to visualize them).\n\n![Hybrid Filled Density](docs/assets/media/squadv1/models/network_filled/density_info.png)\n\nYou can see here the pruned heads for each layer:\n\n![Hybrid Filled Head Pruning](docs/assets/media/squadv1/models/network_filled/pruning_info.png)\n\n\n### Comparison with state of the art\nIf we plot the F1 of the full set of pruned networks against the speedup, we can see that we outperform fine-tuned TinyBERT and Distilbert by some margin.\nMobileBert seems significantly better, even with the \"no OPT\" version presented here, which does not contain the LayerNorm optimization used in the much faster version of MobileBERT.\nAn interesting future work will be to add those optimizations to the pruning tools.\n\n![SQuAD v1 speedup](docs/assets/media/squadv1/graphs/summary_speedup.png)\n\nEven in terms of saved size, we get smaller networks for the same accuracy (except for MobileBERT, which is better on size too):\n\n![SQuAD fill rate](docs/assets/media/squadv1/graphs/summary_fill_rate.png)\n\n### GLUE/MNLI\n\nThe experiments were done on BERT-base.\nSignificant speedups were obtained, even if the results are a bit behind compared to the SQuAD results.\nHere is a selection of networks, with the same rules as for the SQuAd table:\n\n|                         Model                          |  Type  |   method    |Params |Accuracy|Accuracy diff| Speedup |\n|--------------------------------------------------------|--------|-------------|-------|--------|-------------|---------|\n|**[#1](https://huggingface.co/aloxatel/bert-base-mnli)**|**base**|**-**        |**+0%**|**84.6**|**+0.00**    |**1.00x**|\n|#2                                                      |base    |hybrid-filled|-65%   |   83.71|-0.89        |2.00x    |\n|#3                                                      |base    |hybrid-filled|-74%   |   83.05|-1.55        |2.40x    |\n|#4                                                      |base    |hybrid-filled|-81%   |   82.69|-1.91        |2.86x    |\n|#5                                                      |base    |hybrid-filled|-87%   |   81.03|-3.57        |3.44x    |\n\n\n\n\n\n### Comparison with state of the art\n(This is WIP : Some more runs are needed to check the performance versus MobileBERT and TinyBert at same level of speed. Some better hyperparameters may help too.)\n\nFrom the following graphs, we see that the speed is a bit lower compared to TinyBERT, and roughly in line with MobileBERT.\nIn terms of sparsity, the precision is a bit lower than MobileBERT and TinyBERT.\nOn both metrics it's better than DistilBERT by some significant margin.\n\n![MNLI v1 speedup](docs/assets/media/mnli/graphs/summary_speedup.png)\n\n\n![MNLI fill rate](docs/assets/media/mnli/graphs/summary_fill_rate.png)\n\n## Related work\n[pytorch_block_sparse](https://github.com/huggingface/pytorch_block_sparse) is a CUDA Implementation of block sparse kernels for linear layer forward and backward propagation.\nIt's not needed to run the models pruned by the nn_pruning tools, as it's not fast enough yet to be competitive with dense linear layers: just pruning heads is faster, even if those heads still contain some inner sparsity.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuggingface%2Fnn_pruning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhuggingface%2Fnn_pruning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuggingface%2Fnn_pruning/lists"}