{"id":19623451,"url":"https://github.com/airaria/textpruner","last_synced_at":"2025-04-05T20:06:19.844Z","repository":{"id":40342131,"uuid":"387739120","full_name":"airaria/TextPruner","owner":"airaria","description":"A PyTorch-based model pruning toolkit for pre-trained language models","archived":false,"fork":false,"pushed_at":"2023-08-31T00:31:15.000Z","size":11013,"stargazers_count":386,"open_issues_count":0,"forks_count":35,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-29T18:09:11.760Z","etag":null,"topics":["model-pruning","pre-trained-language-models","pruning","transformer"],"latest_commit_sha":null,"homepage":"https://textpruner.readthedocs.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/airaria.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-07-20T09:24:06.000Z","updated_at":"2025-03-01T06:18:59.000Z","dependencies_parsed_at":"2024-06-19T03:02:05.894Z","dependency_job_id":"58e6398e-957f-4952-993e-25ee58e9904c","html_url":"https://github.com/airaria/TextPruner","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/airaria%2FTextPruner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/airaria%2FTextPruner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/airaria%2FTextPruner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/airaria%2FTextPruner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/airaria","download_url":"https://codeload.github.com/airaria/TextPruner/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247393569,"owners_count":20931812,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["model-pruning","pre-trained-language-models","pruning","transformer"],"created_at":"2024-11-11T11:33:46.796Z","updated_at":"2025-04-05T20:06:19.826Z","avatar_url":"https://github.com/airaria.png","language":"Python","readme":" [**English**](README.md) | [**中文说明**](README_ZH.md)\n\n\u003cp align=\"center\"\u003e\n    \u003cbr\u003e\n    \u003cimg src=\"./pics/banner.png\" width=\"500\"/\u003e\n    \u003cbr\u003e\n\u003cp\u003e\n\u003cp\u003e\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://github.com/airaria/TextPruner/blob/master/LICENSE\"\u003e\n        \u003cimg alt=\"GitHub\" src=\"https://img.shields.io/github/license/airaria/TextPruner.svg?color=green\u0026style=flat-square\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://TextPruner.readthedocs.io/\"\u003e\n        \u003cimg alt=\"Documentation\" src=\"https://img.shields.io/website?down_message=offline\u0026label=Documentation\u0026up_message=online\u0026url=https%3A%2F%2FTextPruner.readthedocs.io\"\u003e\n    \u003c/a\u003e    \n    \u003ca href=\"https://pypi.org/project/TextPruner\"\u003e\n        \u003cimg alt=\"PyPI\" src=\"https://img.shields.io/pypi/v/TextPruner\"\u003e\n    \u003c/a\u003e    \n    \u003ca href=\"https://github.com/airaria/TextPruner/releases\"\u003e\n        \u003cimg alt=\"GitHub release\" src=\"https://img.shields.io/github/v/release/airaria/TextPruner?include_prereleases\"\u003e\n    \u003c/a\u003e\n\u003c/p\u003e\n\n**TextPruner** is a model pruning toolkit for pre-trained language models.\nIt provides **low-cost** and **training-free** methods to reduce your model size and speed up your model inference speed by removing redundant neurons.\n\nYou may also be interested in,\n\n- Knowledge Distillation Toolkit - TextBrewer: https://github.com/airaria/TextBrewer\n- Chinese MacBERT: https://github.com/ymcui/MacBERT\n- Chinese ELECTRA: https://github.com/ymcui/Chinese-ELECTRA\n- Chinese XLNet: https://github.com/ymcui/Chinese-XLNet\n- CINO: https://github.com/ymcui/Chinese-Minority-PLM\n\n## News\n\n* [Mar 21, 2022] （new functionality in v1.1) Added vocabulary pruning for XLM, BART, T5 and mT5 models.\n\n* [Mar 4, 2022] We are delighted to announce that TextPruner paper [*TextPruner: A Model Pruning Toolkit for Pre-Trained Language Models*](https://aclanthology.org/2022.acl-demo.4/) has been accepted to [ACL 2022 demo](https://2022.aclweb.org).\n\n* [Jan 26, 2022] （new functionality in v1.0.1) Added support for self-supervised pruning via `use_logits` option in `TransformerPruningConfig`.\n\n## Table of Contents\n\n\u003c!-- TOC --\u003e\n\n| Section | Contents |\n|-|-|\n| [Introduction](#introduction) | Introduction to TextPruner |\n| [Installation](#installation) | Requirements and how to install |\n| [Pruning Modes](#pruning-modes) | A brief introduction to the three pruning modes |\n| [Usage](#usage) | A quick guide on how to use TextPruner |\n| [Experiments](#experiments) | Pruning experiments on typical tasks |\n| [FAQ](#faq) | Frequently asked questions |\n| [Follow Us](#follow-us) | - |\n\n\n\n## Introduction\n\n**TextPruner** is a toolkit for pruning pre-trained transformer-based language models written in PyTorch. It offers structured training-free pruning methods and a user-friendly interface.\n\nThe main features of TexPruner include:\n\n* **Compatibility**: TextPruner is compatible with different NLU pre-trained models. You can use it to prune your own models for various NLP tasks as long as they are built on the standard pre-trained models.\n* **Usability**: TextPruner can be used as a package or a CLI tool. They are both easy to use.\n* **Efficiency**: TextPruner reduces the model size in a simple and fast way. TextPruner uses structured training-free methods to prune models. It is much faster than distillation and other pruning methods that involve training. \n\nTextPruner currently supports vocabulary pruning and transformer pruning. For the explanation of the pruning modes, see [Pruning Modes](#pruning-modes).\n\nTo use TextPruner, users can either import TextPruner into the python scripts or run the TextPruner command line tool. See the examples in [Usage](#usage).\n\nFor the performance of the pruned model on typical tasks, see [Experiments](#experiments).\n\nPaper: [*TextPruner: A Model Pruning Toolkit for Pre-Trained Language Models*](https://aclanthology.org/2022.acl-demo.4/)\n\n\n### Supporting Models\n\nTextPruner currently supports the following pre-trained models in [transformers](https://github.com/huggingface/transformers):\n\n| Model | Vocabualry Pruning | Transformer Pruning |\n|-|:-:|:-:|\n| BERT | :heavy_check_mark: | :heavy_check_mark: |  \n| ALBERT | :heavy_check_mark: | :heavy_check_mark: |  \n| RoBERTa | :heavy_check_mark: | :heavy_check_mark: |  \n| ELECTRA| :heavy_check_mark: | :heavy_check_mark: |  \n| XLM-RoBERTa | :heavy_check_mark: | :heavy_check_mark: |\n|XLM |  :heavy_check_mark: | :x: |\n|BART |  :heavy_check_mark: | :x: |\n|T5 |  :heavy_check_mark: | :x: |\n|mT5 |  :heavy_check_mark: | :x: |\n\n\nSee the [online documentation](https://textpruner.readthedocs.io) for the API reference.\n\n\n## Installation\n\n* Requirements\n\n    * Python \u003e= 3.7\n    * torch \u003e= 1.7\n    * transformers \u003e= 4.0\n    * sentencepiece\n    * protobuf\n\n* Install with pip\n\n    ```bash\n    pip install textpruner\n    ```\n\n*  Install from the source\n\n    ```bash\n    git clone https://github.com/airaria/TextPruner.git\n    pip install ./textpruner\n    ```\n\n### Pruning Modes\n\nIn TextPruner, there are three pruning modes: **vocabulary pruning**, **transformer pruning** and **pipeline pruning**.\n\n\n![](pics/PruningModes.png)\n\n#### Vocabulary Pruning\n\nThe pre-trained models usually have a large vocabulary, but some tokens rarely appear in the datasets of the downstream tasks. These tokens can be removed to reduce the model size and accelerate MLM pre-training.\n\n#### Transformer Pruning\nAP\n\nAnother approach is pruning the transformer blocks. Some studies have shown that not all attention heads are equally important in the transformers. TextPruner reduces the model size and keeps the model performance as high as possible by locating and removing the unimportant attention heads and the feed-forward networks' neurons.\n\n#### Pipeline Pruning\n\n\nIn pipeline pruning, TextPruner performs transformer pruning and vocabulary pruning successively to fully reduce the model size.\n\n\n## Usage\n\nThe **pruners** perform the pruning process. The **configurations** set their behaviors. There names are self-explained: \n* Pruners\n  * `textpruner.VocabularyPruner`\n  * `textpruner.TransformerPruner`\n  * `textpruner.PipelinePruner`\n* Configurations\n  * `textpruner.GeneralConfig`\n  * `textpruner.VocabularyPruningConfig`\n  * `textpruner.TransformerPruningConfig`\n\nSee the [online documentation](https://textpruner.readthedocs.io) for the API reference.\nThe `Configurations` are explained in [Configurations](#configurations).\nWe demonstrate the basic usage below.\n\n### Vocabulary Pruning\n\nTo perform vocabulary pruning, users should provide a text file or a list of strings. The tokens that do not appear in the texts are removed from the model and the tokenizer.\n\nSee the examples at [examples/vocabulary_pruning](examples/vocabulary_pruning) and [examples/vocabulary_pruning_xnli](examples/vocabulary_pruning_xnli).\n\n#### Use TextPruner as a package\n\nPruning the vocabulary in 3 lines of code:\n\n```python\nfrom textpruner import VocabularyPruner\npruner = VocabularyPruner(model, tokenizer)\npruner.prune(dataiter=texts)\n```\n\n*  `model` is the pre-trained model for the MLM task or other NLP tasks.\n* `tokenizer` is the corresponding tokenizer.\n* `texts` is a list of strings. The tokens that do not appear in the texts are removed from the model and the tokenizer.\n\n\n`VocabularyPruner` accepts `GeneralConfig` and `VocabularyPruningConfig` for fine control. By default we could omit them. See the API reference for details.\n\n\n#### Use TextPruner-CLI tool\n\n```bash\ntextpruner-cli  \\\n  --pruning_mode vocabulary \\\n  --configurations gc.json vc.json \\\n  --model_class XLMRobertaForSequenceClassification \\\n  --tokenizer_class XLMRobertaTokenizer \\\n  --model_path /path/to/model/and/config/directory \\\n  --vocabulary /path/to/a/text/file\n```\n* `configurations` : configuration files in the JSON format. See [Configurations](#configurations) for details.\n* `model_class` : The classname of the model. It must be accessible from the current directory. For example, if `model_class` is `modeling.ModelClassName`, there should be a `modeling.py` in the current directory. If there is no module name in `model_class`, TextPruner will try to import the `model_class` from the transformers library, as shown above.\n* `tokenizer_class` : The classname of the tokenizer. It must be accessible from the current directory. If there is no module name in `tokenizer_class`, TextPruner will try to import the `tokenizer_class` from the transformers library.\n* `model_path` : The directory that contains weight and the configurations for the model and the tokenizer.\n* `vocabulary` : A text file that is used for generating new vocabulary. The tokens that do not appear in the vocabulary are removed from the model and the tokenizer.\n\n\n### Transformer Pruning\n\n* To perform transformer pruning on a dataset, a `dataloader` of the dataset should be provided. The `dataloader` should return both the inputs and the labels. \n* TextPruner needs the loss returned by the model to calculate neuron importance scores. TextPruner will try to guess which element in the model output is the loss. If none of the following is true:\n  * the model returns  a single element, which is the loss;\n  * the model output is a list or a tuple. Loss is its first element;\n  * the loss of can be accessed by `output['loss'] ` or `output.loss` where `output` is the model output\n\n  users should provide an `adaptor` function (which takes the output of the model and return the loss) to the `TransformerPruner`.\n\n  * If running in *self-supervised* mode, TextPruner needs the logits returned by the model to calculate importance scores. In this case,  the `adaptor` should return the logits. Check the `use_logits` option in `TransformerPruningConfig` for details.\n\nSee the examples at [examples/transformer_pruning](examples/transformer_pruning).\n\nFor self-supervised pruning, see the examples [examples/transformer_pruning_xnli](examples/transformer_pruning_xnli).\n\n#### Use TextPruner as a package\n\n```python\nfrom textpruner import TransformerPruner, TransformerPruningConfig\ntransformer_pruning_config = TransformerPruningConfig(\n      target_ffn_size=2048, \n      target_num_of_heads=8, \n      pruning_method='iterative',\n      n_iters=4)\npruner = TransformerPruner(model,transformer_pruning_config=transformer_pruning_config)   \npruner.prune(dataloader=dataloader, save_model=True)\n```\n\n*  `transformer_pruning_config` set the mean target size per layer (`target_ffn_size` and `target_num_of_heads`) and the number of iterations (`n_iters`) of pruning.\n* `dataloader` is a PyTorch dataloader that provides inputs and labels of the dataset.\n\n`TransformerPruner` accepts `GeneralConfig` and `TransformerPruningConfig` for fine control. See the API reference for details.\n\n\n#### Use TextPruner-CLI tool\n\n```bash\ntextpruner-cli  \\\n  --pruning_mode transformer \\\n  --configurations gc.json tc.json \\\n  --model_class XLMRobertaForSequenceClassification \\\n  --tokenizer_class XLMRobertaTokenizer \\\n  --model_path ../models/xlmr_pawsx \\\n  --dataloader_and_adaptor dataloader_script\n```\n* `configurations` : configuration files in the JSON format. See [Configurations](#configurations) for details.\n* `model_class` : The classname of the model. It must be accessible from the current directory. For example, if `model_class` is `modeling.ModelClassName`, there should be a `modeling.py` in the current directory. If there is no module name in `model_class`, TextPruner will try to import the `model_class` from the transformers library, as shown above.\n* `tokenizer_class` : The classname of the tokenizer. It must be accessible from the current directory. If there is no module name in `tokenizer_class`, TextPruner will try to import the `tokenizer_class` from the transformers library.\n* `model_path` : The directory contains weight and the configurations for the model and the tokenizer.\n* `dataloader_and_adaptor` : The python script that contains the dataloader and the adaptor (the adaptor is optional).\n\n\n\n### Pipeline Pruning\n\nPipeline pruning combines transformer pruning and vocabulary pruning into a single call.\n\nSee the examples at [examples/pipeline_pruning](examples/pipeline_pruning).\n\n#### Use TextPruner as a package\n\n```python\nfrom textpruner import PipelinePruner, TransformerPruningConfig\ntransformer_pruning_config = TransformerPruningConfig(\n    target_ffn_size=2048, target_num_of_heads=8, \n    pruning_method='iterative',n_iters=4)\npruner = PipelinePruner(model, tokenizer, transformer_pruning_config=transformer_pruning_config)\npruner.prune(dataloader=dataloader, dataiter=texts, save_model=True)\n```\n\n`PipelinePruner` accepts `GeneralConfig`, `VocabularyPruningConfig` and `TransformerPruningConfig` for fine control. See the API reference for details.\n\n#### Use TextPruner-CLI tool\n\n```bash\ntextpruner-cli  \\\n  --pruning_mode pipeline \\\n  --configurations gc.json tc.json vc.json \\\n  --model_class XLMRobertaForSequenceClassification \\\n  --tokenizer_class XLMRobertaTokenizer \\\n  --model_path ../models/xlmr_pawsx \\\n  --vocabulary /path/to/a/text/file \\\n  --dataloader_and_adaptor dataloader_script\n```\n\n### Configurations\n\nThe pruning process is configured by the configuration objects:\n\n* `GeneralConfig` : sets the device and the output directory.\n* `VocabularyPruningConfig` : sets the token pruning threshold and whether pruning the `lm_head`.\n* `TransformerPruningConfig` : sets various options on how to perform the transformer pruning process.\n\nThey are used in different pruning modes:\n\n* Vocabulary pruning accepts `GeneralConfig` and `VocabularyPruningConfig`\n\n  ```python\n  VocabularyPruner(vocabulary_pruning_config= ..., general_config = ...)\n  ```\n\n* Transformer pruning accepts `GeneralConfig` and `TransformerPruningConfig`\n  ```python\n  TransformerPruner(transformer_pruning_config= ..., general_config = ...)\n  ```\n\n* Pipeline pruning accepts all the configurations\n  ```python\n  TransformerPruner(transformer_pruning_config= ..., vocabulary_pruning_config= ..., general_config = ...)\n  ```\n\nThe configurations are dataclass objects (used in the python scripts) or JSON files (used in the command line).\nIf no configurations are provided, TextPruner will use the default configurations.\nSee the API reference for details.\n\n\nIn the python script:\n\n```python\nfrom textpruner import GeneralConfig, VocabularyPruningConfig, TransformerPruningConfig\nfrom textpruner import VocabularyPruner, TransformerPruner, PipelinePruner\n\n#GeneralConfig\ngeneral_config = GeneralConfig(device='auto',output_dir='./pruned_models')\n\n#VocabularyPruningConfig\nvocabulary_pruning_config = VocabularyPruningConfig(min_count=1,prune_lm_head='auto')\n\n#TransformerPruningConfig\n#Pruning with the given masks \ntransformer_pruning_config = TransformerPruningConfig(pruning_method = 'masks')\n\n#TransformerPruningConfig\n#Pruning on labeled dataset iteratively\ntransformer_pruning_config = TransformerPruningConfig(\n    target_ffn_size  = 2048,\n    target_num_of_heads = 8,\n    pruning_method = 'iterative',\n    ffn_even_masking = True,\n    head_even_masking = True,\n    n_iters = 1,\n    multiple_of = 1\n)\n```\n\nAs JSON files:\n\n* `GeneralConfig` : [gc.json](examples/configurations/gc.json)\n* `VocabularyPruningConfig` : [vc.json](examples/configurations/vc.json)\n* `TransformerPruningConfig` :\n    * Pruning with the given masks : [tc-masks.json](examples/configurations/tc-masks.json)\n    * Pruning on labeled dataset iteratively : [tc-iterative.json](examples/configurations/tc-iterative.json)\n\n\n### Helper functions\n\n* `textpruner.summary` : show the summary of model parameters.\n* `textpruner.inference_time` : measure and print the inference time of the model.\n\nExample:\n\n```python\nfrom transformers import BertForMaskedLM\nimport textpruner\nimport torch\n\nmodel = BertForMaskedLM.from_pretrained('bert-base-uncased')\nprint(\"Model summary:\")\nprint(textpruner.summary(model,max_level=3))\n\ndummy_inputs = [torch.randint(low=0,high=10000,size=(32,512))]\nprint(\"Inference time:\")\ntextpruner.inference_time(model.to('cuda'),dummy_inputs)\n```\n\nOutputs:\n\n```\nModel summary:\nLAYER NAME                          \t        #PARAMS\t     RATIO\t MEM(MB)\n--model:                            \t    109,514,810\t   100.00%\t  417.77\n  --bert:                           \t    108,892,160\t    99.43%\t  415.39\n    --embeddings:                   \t     23,837,696\t    21.77%\t   90.94\n      --position_ids:               \t            512\t     0.00%\t    0.00\n      --word_embeddings:            \t     23,440,896\t    21.40%\t   89.42\n      --position_embeddings:        \t        393,216\t     0.36%\t    1.50\n      --token_type_embeddings:      \t          1,536\t     0.00%\t    0.01\n      --LayerNorm:                  \t          1,536\t     0.00%\t    0.01\n    --encoder\n      --layer:                      \t     85,054,464\t    77.66%\t  324.46\n  --cls\n    --predictions(partially shared):\t        622,650\t     0.57%\t    2.38\n      --bias:                       \t         30,522\t     0.03%\t    0.12\n      --transform:                  \t        592,128\t     0.54%\t    2.26\n      --decoder(shared):            \t              0\t     0.00%\t    0.00\n\nInference time:\nDevice: cuda:0\nMean inference time: 1214.41ms\nStandard deviation: 2.39ms\n```\n\n\n## Experiments\n\n\nWe prune a [XLM-RoBERTa-base](https://github.com/facebookresearch/XLM) classification model trained on the Multilingual Natural Language Inference (NLI) task [PAWS-X](https://github.com/google-research-datasets/paws/tree/master/pawsx). The model is fine-tuned and evaluated on the Egnlish dataset.\n\n### Vocabulary Pruning\nWe use a [100k-lines subset](examples/datasets/xnli/en.tsv) of [XNLI](https://github.com/facebookresearch/XNLI) English training set as the vocabulary file. The pruning result is listed below.\n\n| Model                | Total size (MB) | Vocab size | Acc on en (%)|\n| :-------------------- | :---------------: | :----------: | :------------: |\n| XLM-RoBERTa-base     | 1060 (100%)     | 250002     | 94.65        |\n| + Vocabulary Pruning | 398 (37.5%)     | 23936      | 94.20        |\n\n### Transfomer Pruning\n\nWe denote the model structure as `(H, F)` where `H` is the average number of attention heads per layer, `F` is the average FFN hidden size per layer. With this notation, `(12,3072)` stands for the original XLM-RoBERTa model. In addition we consider (8, 2048) and (6, 1536).\n\n#### Inference time \n\nThe speed is measured on inputs of length 512 and batch size 32. \nEach layer of the model has the same number of attention heads and FFN hidden size.\n\n| Model      | Total size (MB) | Encoder size (MB) | Inference time (ms) | Speed up |\n| :---------- | :---------------: | :-----------------: | :-------------------: | :--------: |\n| (12, 3072) | 1060            | 324               | 1012                | 1.0x     |\n| (8, 2048)  | 952             | 216               | 666                 | 1.5x     |\n| (6, 1536)  | 899             | 162               | 504                 | 2.0x     |\n\n\n#### Performance\n\nWe prune the model with different numbers of iterations (`n_iters`). The accuracies are listed below:\n\n| Model      | n_iters=1           |           n_iters=2 |           n_iters=4 |           n_iters=8 |           n_iters=16 |\n| :------------ | :-----------: | :-----------: | :-----------: | :-----------: | :------------: |\n| (12, 3072)   | 94.65       | -           | -           | -           | -            |\n| (8, 2048)    | 93.30       | 93.60       | 93.60       | 93.85       | 93.95        |\n| (8, 2048) with uneven heads   | 92.95       | 93.50       | 93.95       | 94.05        | **94.25**    |\n| (6, 1536)    | 85.15       | 89.10       | 90.90       | 90.60       | 90.85        |\n| (6, 1536) with uneven heads   | 45.35       | 86.45       |  90.55     | 90.90         | **91.95**    |\n\n`uneven heads` means the number of attention heads may vary from layer to layer.\nWith the same model structure, the performance increases as we increase the number of iterations `n_iters`. \n\n## FAQ\n\n**Q: Does TextPruner support Tensorflow 2 ?**\n\nA: No. \n\n**Q: Can you compare the knowledge distillation and model pruning? Which one should I use ?**\n\nA: Both model pruning and knowledge distillation are popular approaches for reducing the model size and accelerating model speed. \n\n* Knowledge distillation usually achieves better performance and a higher compression ratio, but the distillation process is computationally expensive and time-costing. It requires accessing a large amount of data for training.\n\n* The structured training-free pruning usually leads to a lower performance than knowledge distillation, but the method is fast and light. The pruning process can be finished within minutes, and only requires a small amount of data for guiding the pruning process.\n\n(There are some pruning methods that involves training can also achieve a high compression ratio)\n\nIf you are interested in applying knowledge distillation, please refer to our [TextBrewer](http://textbrewer.hfl-rc.com).\n\nif you want to achieve the best performance, you may consider applying both distillation and pruning.\n\n## Citation\nIf you find TextPruner is helpful, please cite our [paper](https://aclanthology.org/2022.acl-demo.4):\n```\n@inproceedings{yang-etal-2022-textpruner,\n    title = \"{T}ext{P}runer: A Model Pruning Toolkit for Pre-Trained Language Models\",\n    author = \"Yang, Ziqing  and\n      Cui, Yiming  and\n      Chen, Zhigang\",\n    booktitle = \"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations\",\n    month = may,\n    year = \"2022\",\n    address = \"Dublin, Ireland\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2022.acl-demo.4\",\n    pages = \"35--43\"\n}\n```\n\n## Follow Us\nFollow our official WeChat account to keep updated with our latest technologies!\n\n![](pics/hfl_qrcode.jpg)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fairaria%2Ftextpruner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fairaria%2Ftextpruner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fairaria%2Ftextpruner/lists"}