{"id":18264546,"url":"https://github.com/modeltc/outlier_suppression_plus","last_synced_at":"2025-04-04T21:30:36.325Z","repository":{"id":171867277,"uuid":"647645553","full_name":"ModelTC/Outlier_Suppression_Plus","owner":"ModelTC","description":"Official implementation of the EMNLP23 paper: Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling","archived":false,"fork":false,"pushed_at":"2023-10-21T20:55:56.000Z","size":810,"stargazers_count":47,"open_issues_count":0,"forks_count":4,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-03-20T19:15:35.339Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ModelTC.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-31T08:17:21.000Z","updated_at":"2025-02-12T01:40:21.000Z","dependencies_parsed_at":null,"dependency_job_id":"24126fbc-a66b-4420-aa97-9f6a49431754","html_url":"https://github.com/ModelTC/Outlier_Suppression_Plus","commit_stats":null,"previous_names":["modeltc/outlier_suppression_plus"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ModelTC%2FOutlier_Suppression_Plus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ModelTC%2FOutlier_Suppression_Plus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ModelTC%2FOutlier_Suppression_Plus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ModelTC%2FOutlier_Suppression_Plus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ModelTC","download_url":"https://codeload.github.com/ModelTC/Outlier_Suppression_Plus/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247251971,"owners_count":20908600,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-05T11:15:04.682Z","updated_at":"2025-04-04T21:30:35.469Z","avatar_url":"https://github.com/ModelTC.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Outlier Suppression+\n\nOfficial PyTorch implementation of  \u003ca href=\"https://arxiv.org/abs/2304.09145\"\u003eOutlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling\u003c/a\u003e\n\n## Overview\n\nThe Outlier Suppression+ (OS+) effectively suppresses outliers in large language models for better quantization performance without extra inference burden. It first identifies the outlier asymmetric shape across channels and proposes a channel-wise shifting technique with a migration pattern to eliminate it. It then focuses on the outlier concentration phenomenon and proposes to scale down outlier channels toward an elaborate objective. \n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"figure/outlier_suppression_plus.png\"\u003e\n\u003c/p\u003e\n\nWe assess the efficacy of our approach under both standard and fine-grained quantization settings. On standard one, OS+ achieves near-floating-point performance on 8-bit and 6-bit BERT, OPTs, BLOOM, and BLOOMZ. On fine-grained one, OS+ can surpass others by 9.41\\% on 4-bit LLaMA with per-token quantization and obtain lossless results on 4-bit OPT with per-group quantization. \n\nIn the following sections, [Support](#support) gives supported models and quantization schemes, [Getting Started](#getting started) introduces the whole procedure to run this project including data preparation, quantization, evaluation and updated model export. [Evaluation](#evaluation) lists configs for each table in the paper for other researchers to reproduce.\n\n## Support\n\n* Models: detailed analysis of outliers can be found in [outlier_analysis.md](outlier_analysis.md)\n  - [x] OPTs\n  - [x] BLOOM, BLOOMZ\n  - [x] LLaMA\n\n* Quantization scheme\n  - [x] Standard quantization (per-channel vs per-tensor, symmetric vs asymmetric)\n  - [x] Per-token quantization proposed in [ZeroQuant](https://arxiv.org/abs/2206.01861)\n  - [x] Per-group quantization proposed in [ZeroQuant-V2](https://arxiv.org/abs/2303.08302)\n\n* Bits\n  - [x] 8-bit\n  - [x] 6-bit\n  - [x] 4-bit\n\n## Getting Started\n\n### Preparation\n\n* Prepare A100 to run the code.\n\n* Install transformers (version 4.30.0) and datasets (version 2.13.1). \n\n* Prepare pre-trained models such as OPTs, BLOOMs.\n\n* For codes of FP tasks, we adopt the implementation in \u003ca href=\"https://github.com/EleutherAI/lm-evaluation-harness\"\u003elm-evaluation-harness\u003c/a\u003e. Thus, download your dataset and update your own data path in lm_eval/dataset directory for later evaluation.\n\n* Considering quantization on zero-shot tasks here, additionally prepare the pile datasets (part of train datasets) for later calibration. Please first download the PILE dataset to ${pile_path}, and run the following code which will randomly select 128 samples for calibration. \n\n  ```\n  dataset_path = ${pile_path}\n  calibration_size = 128\n  raw_dataset = load_dataset(\"json\", data_files=dataset_path, split=\"train\")\n  random_rows = random.sample(range(raw_dataset.num_rows), calibration_size)\n  raw_dataset = raw_dataset.select(random_rows)\n  raw_dataset.save_to_disk(${pile_cali_path})\n  ```\n\n### Quantization\n\nConduct quantization on specific models and with certain quantization schemes. Here is a simple example for OPT-66B and 8-bit per-tensor symmetric quantization.\n\n```\n# opt.sh\nmodel_size=66b\ntask_name=winogrande\nmodel_path=model_path\nq_config=exp/opt/int8.yaml # q_config.yaml path\nis_quant=True # True: quantization; False: fp16\n\nexport CUDA_VISIBLE_DEVICES=0,1,2,3\npython main.py \\\n    --model opt \\\n    --model_args pretrained=$model_path \\\n    --tasks $task_name \\\n    --batch_size 16 \\\n    --no_cache \\\n    --dtype 'float16' \\\n    --is_quant ${is_quant} \\\n    --config ${q_config} \\\n    2\u003e\u00261 | tee experiment/opt_${model_size}_${task_name}.log\n```\n\nModel is assigned with --model and --model_args pretrained args. Quantization config is assigned with q_config. The task is assigned with task_name.\nBy running the above command, you will get an accuracy of 69.0.\n\n### Export for deployment\n\nAs the method only updates weights and biases of the floating-point model, we can easily export a new FP model with weaker outliers, enjoying convenience for further development. Here is an example to export opt-6.7B.\n\n* Obtain the scaling and shifting values by turning on the \"is_export\" choice in opt.sh file. It will store the calculated values in the ${quant_dir}.\n\n```\n# opt.sh\nmodel_size=6.7b\ntask_name=winogrande\nmodel_path=model_path \nq_config=exp/opt/int8.yaml # q_config.yaml path\nis_quant=True # True: quantization; False: fp16\nexport CUDA_VISIBLE_DEVICES=0\npython main.py \\\n    --model opt \\\n    --model_args pretrained=$model_path \\\n    --tasks $task_name \\\n    --batch_size 16 \\\n    --no_cache \\\n    --dtype 'float16' \\\n    --is_quant ${is_quant} \\\n    --is_export \\\n    --config ${q_config} \\\n    2\u003e\u00261 | tee experiment/opt_${model_size}_${task_name}.log\n```\n\n* Use them to update the original FP model by running export.sh\n\n```\n# export.sh\nmodel_size=6.7b   # model size\nmodel_type=opt    # model type\nmodel_path=model_path   # original model path\noutput_path=output_path # new FP model path\nscale_shift_list=exp/opt/scale_shift_list.pth # scaling and shifting values path\n\nexport CUDA_VISIBLE_DEVICES=0\npython quant_transformer/solver/export.py \\\n    --model_path $model_path \\\n    --scale_shift_list $scale_shift_list \\\n    --model_type $model_type \\\n    --output_path $output_path\n```\n\n## Evaluation\n\n### Introduction of config.yaml\n\n```\nquant: \n    a_qconfig: # quantization details for activation\n        quantizer: FixedFakeQuantize  # quantizer type\n        observer: AvgMinMaxObserver  # calibration methods\n        bit: 8 # bit selection\n        symmetric: True  # True: symmetric quantization, False: asymmetric one\n        ch_axis: -1  # -1: per-layer quantization\n    w_qconfig: # quantization details for weight\n        quantizer: FixedQuantize # Quantizer type\n        observer: MinMaxObserver # calibration methods\n        bit: 8 # bit selection\n        symmetric: True # True: symmetric quantization, False: asymmetric one\n        ch_axis: -1  # 0: per-channel quantization, -1: per-layer one\n    calibrate: 128 # calibration size\n    calibrate_path: /mnt/lustre/weixiuying.vendor/datasets/nlp_datasets/pile_cali # calibration dataset path, make sure there is _cali in the name\n\t  except_quantizer: null\n    is_remove_padding: True # True: remove [PAD] during calibration\n    migrate: True # True: shifting and scaling operations, False: no shifting and scaling operations.\n model:\n    max_length: 512 # For PIQA, Winogrande tasks, 512 is enough. For WikiText2, a longer one can improve FP16 results.\n```\n\n### Standard quantization with OS+ \n\n* **8-bit and 6-bit OPTs (Table 2 and Table 9)**\n\n  opt.sh:\n\n    ```\n  model_size=66b\n  task_name=winogrande\n  model_path=model_path\n  q_config=exp/opt/int8.yaml # q_config.yaml path\n  is_quant=True # True: quantization; False: fp16\n  \n  export CUDA_VISIBLE_DEVICES=0,1,2,3\n  python main.py \\\n  --model opt \\\n  --model_args pretrained=$model_path \\\n  --tasks $task_name \\\n  --batch_size 16 \\\n  --no_cache \\\n  --dtype 'float16' \\\n  --is_quant ${is_quant} \\\n  --config ${q_config} \\\n  2\u003e\u00261 | tee experiment/opt_${model_size}_${task_name}.log\n    ```\n\n  q_config:\n  \texp/opt/int8.yaml\n      exp/opt/int6.yaml\n\n* **8-bit and 6-bit BLOOM and BLOOMZ (Table 2)**\n\n  bloom.sh:\n\n    ```\n  model_size=176b\n  task_name=winogrande\n  model_path=model_path \n  q_config=exp/bloom/int8.yaml # quantization config path\n  is_quant=True # True: quantization; False: fp16\n  \n  export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7\n  python main.py \\\n  --model bloom \\\n  --model_args pretrained=$model_path \\\n  --tasks ${task_name} \\\n  --batch_size 12 \\\n  --no_cache \\\n  --dtype 'float16' \\\n  --is_quant ${is_quant} \\\n  --config ${q_config} \\\n  2\u003e\u00261 | tee experiment/bloom_${model_size}_${task_name}.log\n    ```\n\n  q_config:\n  \texp/bloom/int8.yaml\n  \texp/bloom/int6.yaml\n  \texp/bloomz/int8.yaml\n  \texp/bloomz/int6.yaml\n\n### Fine-grained quantization with OS+\n\nHere, OS+ is combined with fine-grained quantization to validate its wide application and go extremely low bit setting like 4-bit quantization.\n\n* **6-bit and 4-bit LLaMA with per-token quantization (Table 3 and Table 10)**\n\n  Per-token quantization which customizes quantization parameters for individual tokens, can bring better predictions, especially for lower-bit quantization and longer output like WikiText2.\n\n  llama.sh:\n\n    ```\n  model_size=7b\n  task_name=piqa\n  model_path=model_path \n  q_config=exp/llama/int6_token.yaml # quantization config path\n  is_quant=True # True: quantization; False: fp16\n  \n  export CUDA_VISIBLE_DEVICES=0\n  python main.py \\\n  --model llama \\\n  --model_args pretrained=${model_path} \\\n  --tasks ${task_name} \\ \n  --batch_size 16 \\\n  --no_cache \\\n  --dtype 'bfloat16' \\\n  --is_quant ${is_quant} \\ \n  --config ${q_config}\"\n  2\u003e\u00261 | tee experiment/llama_${model_size}_${task_name}.log\n    ```\n\n  q_config:\n  \texp/llama/int6_token_disable.yaml\n  \texp/llama/int4_token_disable.yaml\n  \texp/llama/int6_token.yaml\n  \texp/llama/int4_token.yaml\n\n  \n\n* **4-bit OPT with per-group quantization (Figure 3)**\n\n  opt.sh: \n\n  ​\tthe same with the one in standard quantization\n\n  q_config: \n\n  ​\texp/opt/int4_group.yaml\n\n## Reference\n\nIf you find this repo useful for your research, please consider citing the paper:\n\n```\n@article{wei2023outlier,\n    title={Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling},\n    author={Wei, Xiuying and Zhang, Yunchen and Li, Yuhang and Zhang, Xiangguo and Gong, Ruihao and Guo, Jinyang and Liu, Xianglong},\n    journal={arXiv preprint arXiv:2304.09145},\n    year={2023}\n    }\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmodeltc%2Foutlier_suppression_plus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmodeltc%2Foutlier_suppression_plus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmodeltc%2Foutlier_suppression_plus/lists"}