{"id":13753971,"url":"https://github.com/hahnyuan/PB-LLM","last_synced_at":"2025-05-09T21:36:19.798Z","repository":{"id":198816312,"uuid":"656475263","full_name":"hahnyuan/PB-LLM","owner":"hahnyuan","description":"PB-LLM: Partially Binarized Large Language Models","archived":false,"fork":false,"pushed_at":"2023-11-20T12:34:57.000Z","size":21715,"stargazers_count":141,"open_issues_count":5,"forks_count":10,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-08-03T09:06:11.817Z","etag":null,"topics":["neural-networks","quantization"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hahnyuan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-21T03:12:05.000Z","updated_at":"2024-08-03T09:06:15.562Z","dependencies_parsed_at":"2024-08-03T09:17:16.703Z","dependency_job_id":null,"html_url":"https://github.com/hahnyuan/PB-LLM","commit_stats":null,"previous_names":["hahnyuan/pb-llm"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hahnyuan%2FPB-LLM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hahnyuan%2FPB-LLM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hahnyuan%2FPB-LLM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hahnyuan%2FPB-LLM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hahnyuan","download_url":"https://codeload.github.com/hahnyuan/PB-LLM/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224884616,"owners_count":17386121,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["neural-networks","quantization"],"created_at":"2024-08-03T09:01:36.076Z","updated_at":"2024-11-16T06:31:18.160Z","avatar_url":"https://github.com/hahnyuan.png","language":"Python","funding_links":[],"categories":["A01_文本生成_文本对话","Python"],"sub_categories":["大语言对话模型及数据"],"readme":"# PB-LLM: Partially Binarized Large Language Models    \n**[Yuzhang Shang*](https://42shawn.github.io/), [Zhihang Yuan*](http://hahnyuan.com/), Qiang Wu, [Zhen Dong](https://dong-zhen.com/)** (* Equal Contribution)     \n\nThis work explores network binarization, a radical form of quantization, compressing model weights to a single bit, specifically for Large Language Models (LLMs) compression. \nDue to previous binarization methods collapsing LLMs, we propose a novel approach, Partially-Binarized LLM (PB-LLM), which can achieve extreme low-bit quantization while maintaining the linguistic reasoning capacity of quantized LLMs. \nSpecifically, our exploration first uncovers the ineffectiveness of naïve applications of existing binarization algorithms and highlights the imperative role of salient weights in achieving low-bit quantization. \nThus, PB-LLM filters a small ratio of salient weights during binarization, allocating them to higher-bit storage, i.e. partially-binarization. \nPB-LLM is extended to recover the capacities of quantized LMMs, by analyzing from the perspective of post-training quantization (PTQ) and quantization-aware training (QAT). \nUnder PTQ, combining the concepts from GPTQ, we reconstruct the binarized weight matrix guided by the Hessian matrix and successfully recover the reasoning capacity of PB-LLM in low-bit. \nUnder QAT, we freeze the salient weights during training, explore the derivation of optimal scaling factors crucial for minimizing the quantization error, and propose a scaling mechanism based on this derived scaling strategy for residual binarized weights. \nThose explorations and the developed methodologies significantly contribute to rejuvenating the performance of low-bit quantized LLMs and present substantial advancements in the field of network binarization for LLMs. \nThe paper is available at [arxiv](https://arxiv.org/abs/2310.00034).\n\n\n## Tested Models\n\nHuggingface models\n- facebook/opt-125m\n- facebook/opt-1.3b\n- facebook/opt-6.7b\n- huggyllama/llama-7b\n- huggyllama/llama-13b\n\n## Usage\n\n### Environment Setting\n\nIf you use conda, you can create a new environment and install the dependencies with the following commands:\n```shell\nconda create -n binary_llm python=3.10 pip\n```\n\nInstall the python dependencies:\n```shell\npip install torch transformers lm_eval accelerate tensorboardX bitsandbytes sentencepiece\n```\nNote python version must\u003e=3.10\n\n### PTQ (GPTQ-PB)\n\nThe GPTQ-PB is implemented in the [gptq_pb](gptq_pb) folder.\nPlease go to the folder and run the script with the desired arguments:\n```\nusage: run.py [-h] [--plot] [--load_quantized] [--seed SEED] [--nsamples NSAMPLES] [--percdamp PERCDAMP] [--low_frac LOW_FRAC] [--blocksize BLOCKSIZE] [--groupsize GROUPSIZE] [--salient_metric {magnitude,hessian}] [--high_bit HIGH_BIT]\n              [--minlayer MINLAYER] [--maxlayer MAXLAYER] [--quant_only QUANT_ONLY] [--invert] [--save] [--disable_gptq] [--log_wandb]\n              model {wikitext2,ptb,c4} {xnor,sign,no,2bit,4bit,prune}\n\npositional arguments:\n  model                 model to load; for example `huggyllama/llama-7b`.\n  {wikitext2,ptb,c4}    Where to extract calibration data from.\n  {xnor,sign,no,2bit,4bit,prune}\n                        quantization method; `xnor` is the method used in paper; `prune` is the method used in sparseGPTQ\n\n--low_frac LOW_FRAC   fraction of binarized weight\n--salient_metric {magnitude,hessian}    metric to measure salient weights\n```\n\nFor example\n\n```shell\ncd gptq_pb\n# for llama-7b\nCUDA_VISIBLE_DEVICES=1 python run.py huggyllama/llama-7b c4 xnor --low_frac 0.5 --high_bit 8 --salient_metric hessian\nCUDA_VISIBLE_DEVICES=2 python run.py huggyllama/llama-7b c4 xnor --low_frac 0.8 --high_bit 8 --salient_metric hessian\nCUDA_VISIBLE_DEVICES=3 python run.py huggyllama/llama-7b c4 xnor --low_frac 0.9 --high_bit 8 --salient_metric hessian\nCUDA_VISIBLE_DEVICES=0 python run.py huggyllama/llama-7b c4 xnor --low_frac 0.95 --high_bit 8 --salient_metric hessian\n```\n\n### QAT\n\nThe QAT for PB-LLM is implemented in the [qat](qat) folder.\n\nFor example\n\n```shell\n# Testing for debug\nCUDA_VISIBLE_DEVICES='0' python qat/run_qat.py --binarization_method=xnor_outlier --model_id=facebook/opt-125m --train_step=20 --dataset=red_pajama --outlier_fraction 0.1\n# Evaluate\nCUDA_VISIBLE_DEVICES='0' python qat/eval_after_qat.py outputs/facebook/opt-125m/xnor_outlier_0.1_20 --model_id=facebook/opt-125m\n\n\n# for opt-1.3b\nCUDA_VISIBLE_DEVICES='1' python qat/run_qat.py --binarization_method=xnor_outlier --model_id=facebook/opt-1.3b --train_step=10000 --dataset=red_pajama --outlier_fraction 0.1\n# Evaluate\nCUDA_VISIBLE_DEVICES='1' python qat/eval_after_qat.py outputs/facebook/opt-1.3b/xnor_outlier_0.1_10000 --model_id=facebook/opt-1.3b\n\n# hessian based outlier\nCUDA_VISIBLE_DEVICES='2' python qat/run_qat.py --binarization_method=xnor_outlier_hessian --model_id=facebook/opt-1.3b --train_step=10000 --dataset=red_pajama --outlier_fraction 0.1\nCUDA_VISIBLE_DEVICES='2' python qat/eval_after_qat.py outputs/facebook/opt-1.3b/xnor_outlier_hessian_0.1_10000 --model_id=facebook/opt-1.3b\n\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhahnyuan%2FPB-LLM","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhahnyuan%2FPB-LLM","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhahnyuan%2FPB-LLM/lists"}