{"id":13754214,"url":"https://github.com/Aaronhuang-778/BiLLM","last_synced_at":"2025-05-09T22:31:31.429Z","repository":{"id":221454912,"uuid":"747588354","full_name":"Aaronhuang-778/BiLLM","owner":"Aaronhuang-778","description":"[ICML 2024] BiLLM: Pushing the Limit of Post-Training Quantization for LLMs","archived":false,"fork":false,"pushed_at":"2025-01-11T17:16:38.000Z","size":1819,"stargazers_count":207,"open_issues_count":15,"forks_count":14,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-03-08T17:20:14.608Z","etag":null,"topics":["binarization","llm","quantization"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2402.04291","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Aaronhuang-778.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-24T08:37:04.000Z","updated_at":"2025-03-07T12:17:09.000Z","dependencies_parsed_at":"2024-02-08T04:21:12.652Z","dependency_job_id":"2aaca2a9-da20-481a-8f74-f255cd26abeb","html_url":"https://github.com/Aaronhuang-778/BiLLM","commit_stats":null,"previous_names":["aaronhuang-778/billm"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aaronhuang-778%2FBiLLM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aaronhuang-778%2FBiLLM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aaronhuang-778%2FBiLLM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aaronhuang-778%2FBiLLM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Aaronhuang-778","download_url":"https://codeload.github.com/Aaronhuang-778/BiLLM/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253335686,"owners_count":21892713,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["binarization","llm","quantization"],"created_at":"2024-08-03T09:01:50.249Z","updated_at":"2025-05-09T22:31:29.781Z","avatar_url":"https://github.com/Aaronhuang-778.png","language":"Python","funding_links":[],"categories":["A01_文本生成_文本对话"],"sub_categories":["大语言对话模型及数据"],"readme":"# BiLLM: Pushing the Limit of Post-Training Quantization for LLMs [[PDF]](https://arxiv.org/pdf/2402.04291.pdf)\n\n![intuition](imgs/author.png)\n\n**\u003csup\u003e1\u003c/sup\u003eThe University of Hong Kong \u003csup\u003e2 \u003c/sup\u003eBeihang University \u003csup\u003e 3\u003c/sup\u003eETH  Zürich**\n\n![intuition](imgs/main.png)\n\n## Abstract\n\nPretrained large language models (LLMs) exhibit exceptional general language processing capabilities but come with significant demands on memory and computational resources. As a powerful compression technology, binarization can extremely reduce model weights to a mere 1 bit, lowering the expensive computation and memory requirements. However, existing quantization techniques fall short of maintaining LLM performance under ultra-low bit-widths. In response to this challenge, we present *BiLLM*, a groundbreaking 1-bit post-training quantization scheme tailored for pretrained LLMs. Based on the weight distribution of LLMs, *BiLLM* first identifies and structurally selects salient weights, and minimizes the compression loss through an effective *binary residual approximation* strategy. Moreover, considering the bell-shaped distribution of the non-salient weights, we propose an *optimal splitting search* to group and binarize them accurately. *BiLLM* achieving for the first time high-accuracy inference (e.g. 8.41 perplexity on LLaMA2-70B) with only 1.08-bit weights across various LLMs families and evaluation metrics, outperforms SOTA quantization methods of LLM by significant margins. Moreover, *BiLLM* enables the binarization process of the LLM with 7 billion weights within 0.5 hours on a single GPU, demonstrating satisfactory time efficiency.\n\n## News\n\n- [2024/2] *BiLLM* source code is open now!\n\n## Dependencies\n\n* `torch`: tested on v2.0.1+cu117\n* `transformers`: tested on v4.35.0 (the LLaMa integration currently requires a main install from source and `sentencepiece`)\n* `datasets`: tested on v2.14.6\n* `huggingface-hub`: tested on v0.16.4\n\nAll binarization processes and experiments were run on a single 80GB NVIDIA A100. However, all the process can also be conducted on a single 24GB NVIDIA 3090 Ti when the model's parameter is under 70B.\n\n## LLMs Binarization\n\n#### Binarization for OPT families\n\n```\npython3 run.py facebook/opt-6.7b c4 braq --blocksize 128 --salient_metric hessian\n```\n\n\n#### Binarization for LLaMA families\n\n```\npython3 run.py meta-llama/Llama-2-7b-hf c4 braq --blocksize 128 --salient_metric hessian\n```\nor\n```\npython3 run.py huggyllama/llama-7b c4 braq --blocksize 128 --salient_metric hessian\n```\n\n#### Binarization for Vicuna families (Instruction Fine-tuning Models)\n\n```\npython3 run.py lmsys/vicuna-7b-v1.5 c4 braq --blocksize 128 --salient_metric hessian\n```\n\n#### \n\n## Results\n\n- BiLLM  achieve superior perplexity performance on Wikitext2 datasets  within only an average of **1.11** bit-width weights OPT families.\n\n![intuition](imgs/opt_wiki_results.png)\n\n- BiLLM  achieve superior perplexity performance on Wikitext2 datasets  within only an average of **1.09** bit-width weights LLaMA families and **1.08** bit-width weights LLaMA2 families.\n\n![intuition](imgs/llama_wiki_results.png)\n\n- We also evaluated the performance of *BiLLM* on PTB and C4 datasets. \n\n![intuition](imgs/ptb1.png)\n\n![intuition](imgs/ptb2.png)\n\n- We further evaluated *BiLLM* on 7 zero-shot dataset to give extensive insight on  binarization LLMs\n\n  ![intuition](imgs/zero_shot.png)\n\n- BiLLM  achieve superior perplexity performance on Wikitext2 datasets  within only an average of **1.10** bit-width weights Vicuna families (instruction fine-tune models).\n\n![intuition](imgs/vicuna.png)\n\n## Related Project\n[GPTQ: Accurate Post-training Compression for Generative Pretrained Transformers](https://github.com/IST-DASLab/gptq)\n\n[PB-LLM: Partially Binarized Large Language Models](https://github.com/hahnyuan/PB-LLM)\n\n[AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://github.com/mit-han-lab/llm-awq)\n\n\n\n## Citation\n\nIf you find *BiLLM* is useful and helpful to your work, please kindly cite this paper:\n\n```\n@article{huang2024billm,\n  title={BiLLM: Pushing the Limit of Post-Training Quantization for LLMs},\n  author={Huang, Wei and Liu, Yangdong and Qin, Haotong and Li, Ying and Zhang, Shiming and Liu, Xianglong and Magno, Michele and Qi, Xiaojuan},\n  journal={arXiv preprint arXiv:2402.04291},\n  year={2024}\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAaronhuang-778%2FBiLLM","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FAaronhuang-778%2FBiLLM","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAaronhuang-778%2FBiLLM/lists"}