{"id":20464501,"url":"https://github.com/bytedance/abq-llm","last_synced_at":"2025-04-04T13:08:04.558Z","repository":{"id":254807757,"uuid":"846504063","full_name":"bytedance/ABQ-LLM","owner":"bytedance","description":"An acceleration library that supports arbitrary bit-width combinatorial quantization operations","archived":false,"fork":false,"pushed_at":"2024-09-30T13:18:59.000Z","size":56472,"stargazers_count":219,"open_issues_count":14,"forks_count":21,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-03-28T12:05:05.430Z","etag":null,"topics":["cuda","llm-inference","mlsys","quantized-networks","research"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bytedance.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-23T10:55:28.000Z","updated_at":"2025-03-28T08:42:58.000Z","dependencies_parsed_at":"2024-12-31T10:19:42.771Z","dependency_job_id":null,"html_url":"https://github.com/bytedance/ABQ-LLM","commit_stats":null,"previous_names":["bytedance/abq-llm"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytedance%2FABQ-LLM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytedance%2FABQ-LLM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytedance%2FABQ-LLM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytedance%2FABQ-LLM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bytedance","download_url":"https://codeload.github.com/bytedance/ABQ-LLM/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247181632,"owners_count":20897370,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cuda","llm-inference","mlsys","quantized-networks","research"],"created_at":"2024-11-15T13:15:24.413Z","updated_at":"2025-04-04T13:08:04.526Z","avatar_url":"https://github.com/bytedance.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003eABQ-LLM\u003c/h1\n\n\u003cp align=\"center\"\u003eABQ-LLM is a novel arbitrary bit quantization scheme that achieves excellent performance under various quantization settings while enabling efficient arbitrary bit computation at the inference level. \u003c/p\u003e\n\n![ABQ-LLM](fig/ABQ.png)\nThe current release version supports the following features:\n- The ABQ-LLM algorithm is employed for precise weight-only quantization (W8A16, W4A16, W3A16, W2A16) and weight-activation quantization (W8A8, W6A6, W4A4, W3A8, W3A6, W2A8, W2A6).\n- Pre-trained ABQ-LLM model weights for LLM (LLaMA and LLaMA-2 loaded to run quantized models).\n- A set of out-of-the-box arbitrary bit quantization operators that support arbitrary bit model inference in Turing and above architectures.\n\n\n## Contents\n- [Install](#install)\n- [Model Zoo](#abq-llm-model-zoo)\n- [Usage](#usage)\n- [Results](#results)\n- [Citation](#citation)\n\n## Install\n\n### Installation of the algorithmic runtime environment\n```\nconda create -n abq-llm python=3.10.0 -y\nconda activate abq-llm\ngit clone https://github.com/bytedance/ABQ-LLM.git\ncd ./ABQ-LLM/algorithm\npip install --upgrade pip \npip install -r requirements.txt\n```\n\n### Installation of the inference engine environment\nYou can actually compile and test our quantized inference Kernel, but you need to install the basic CUDA Toolkit.\n1. Install CUDA Toolkit (11.8 or 12.1, linux or windows). Use the Express Installation option. Installation may require a restart (windows).\n2. Clone the CUTLASS. (It is only used for speed comparison)\n```\ngit submodule init \ngit submodule update\n```\n\n\n## ABQ-LLM Model\n\nWe provide pre-trained ABQ-LLM model zoo for multiple model families, including LLaMa-1\u00262, OPT.\nThe detailed support list:\n\n| Models  | Sizes                           | W4A16 | W3A16 | W2A16 | W2A16g128 | W2A16g64|\n| ------- | ------------------------------- | ----- | --------- | -------- | ----- |----- |\n| LLaMA   | 7B/13B  | ✅     | ✅         | ✅        | ✅     |✅     |\n| LLaMA-2 | 7B/13B  | ✅     | ✅         | ✅        | ✅     |✅     |\n\n\n| Models       | Sizes                           | W8A8 | W4A8 | W6A6 | W4A6 | W4A4 | W3A8 | W3A6 | W2A8 | W2A6 |\n| ------------ | ------------------------------- | --------- | ----- | --------- | ---- | ---- |---- |---- |---- |---- |\n| LLaMA        | 7B/13B                  | ✅         | ✅     | ✅         | ✅    | ✅    | ✅    | ✅    | ✅    |✅    |\n| LLaMA-2      | 7B/13B                     | ✅         | ✅     | ✅         | ✅    | ✅    | ✅    | ✅    | ✅    |✅    |\n\n\n## Usage\n\n### Algorithm Testing\nWe provide the pre-trained ABQ- LLM model weight in hugginface, you can verify the model performance by the following commands.\n```\nCUDA_VISIBLE_DEVICES=0 python run_pretrain_abq_model.py \\\n--model /PATH/TO/LLaMA/llama-7b-ABQ \\\n--wbits 4 --abits 4\n```\n\nWe also provide full script to run ABQ-LLM in **`./algorithm/scripts/`**. We use LLaMa-7B as an example here:\n1. Obtain the channel-wise scales and shifts required for initialization:\n```\npython generate_act_scale_shift.py --model /PATH/TO/LLaMA/llama-7b\n```\n\n2. Weight-only quantization\n```\n# W3A16\nCUDA_VISIBLE_DEVICES=0 python main.py \\\n--model /PATH/TO/LLaMA/llama-7b  \\\n--epochs 20 --output_dir ./log/llama-7b-w3a16 \\\n--eval_ppl --wbits 3 --abits 16  --lwc --let\n\n# W3A16g128\nCUDA_VISIBLE_DEVICES=0 python main.py \\\n--model /PATH/TO/LLaMA/llama-7b  \\\n--epochs 20 --output_dir ./log/llama-7b-w3a16g128 \\\n--eval_ppl --wbits 3 --abits 16 --group_size 128 --lwc --let\n```\n\n3. weight-activation quantization\n```\n# W4A4\nCUDA_VISIBLE_DEVICES=0 python main.py \\\n--model /PATH/TO/LLaMA/llama-7b  \\\n--epochs 20 --output_dir ./log/llama-7b-w4a4 \\\n--eval_ppl --wbits 4 --abits 4 --lwc --let \\\n--tasks piqa,arc_easy,arc_challenge,boolq,hellaswag,winogrande\n```\n\nMore detailed and optional arguments:\n- `--model`: the local model path or huggingface format.\n- `--wbits`: weight quantization bits.\n- `--abits`: activation quantization bits.\n- `--group_size`: group size of weight quantization. If no set, use per-channel quantization for weight as default.\n- `--lwc`: activate the Learnable Weight Clipping (LWC).\n- `--let`: activate the Learnable Equivalent Transformation (LET).\n- `--lwc_lr`: learning rate of LWC parameters, 1e-2 as default.\n- `--let_lr`: learning rate of LET parameters, 5e-3 as default.\n- `--epochs`: training epochs. You can set it as 0 to evaluate pre-trained ABQ-LLM checkpoints.\n- `--nsamples`: number of calibration samples, 128 as default.\n- `--eval_ppl`: evaluating the perplexity of quantized models.\n- `--tasks`: evaluating zero-shot tasks.\n- `--multigpu`: to inference larger network on multiple GPUs\n- `--real_quant`: real quantization, which can see memory reduce. Note that due to the limitations of AutoGPTQ kernels, the real quantization of weight-only quantization can only lead memory reduction, but with slower inference speed.\n- `--save_dir`: saving the quantization model for further exploration.\n\n### Kernel Benchmark\n1. Compile Kernels.\n\nBy default, w2a2, w3a3, w4a4, w5a5, w6a6, w7a7, w8a8 are compiled, and the kernel of w2a4, w2a6, w2a8, and w4a8 quantization combination is compiled. Each quantization scheme corresponds to dozens of kernel implementation schemes to build its search space.\n```\n# linux\ncd engine\nbash build.sh\n\n# windows\ncd engine\nbuild.bat\n```\n\n\n2. Comprehensive benchmark.\n\nFor the typical GEMM operation of the llama model, different quantization combinations (w2a2, w3a3, w4a4,w5a5, w6a6, w7a7, w8a8, w2a4, w2a6, w2a8, w4a8) are tested to obtain the optimal performance in the search space of each quantization combination.\n```\n# linux\nbash test.sh\n# windows\ntest.bat\n```\n\n3. Add new quantization combinations(Optional).\n\nWe reconstructed the quantized matrix multiplication operation in a clever way, decomposing it into a series of binary matrix multiplications, and performed a high degree of template and computational model abstraction.\n\nBased on the above optimizations, you can quickly expand our code to support new quantization combinations, such as w`p`a`q`. You only need to add w`p`a`q` instantiation definition and declaration files in engine/mma_any/aq_wmma_impl and then recompile.\n\nThe performance upper limit depends on how the search space is defined (the instantiated function configuration). For related experience, please refer to the paper or the existing implementation in this directory.\n\n### E2E Benchmark\n\n1. Compile the fastertransformer\n```\ncd fastertransformer\nbash build.sh\n``` \n\n2. Config llama (Change precision in examples/cpp/llama/llama_config.ini)\n```\nfp16:  int8_mode=0\nw8a16: int8_mode=1\nw8a8:  int8_mode=2\nw4a16: int8_mode=4\nw2a8:  int8_mode=5\n```\n\n3. Run llama on single GPU\n```\ncd build_release\n./bin/llama_example\n```\n\n4. (Optional) Run in multi GPU. Change tensor_para_size=2 in examples/cpp/llama/llama_config.ini\n\n```\ncd build_release\nmpirun -n 2 ./bin/llama_example\n```\n\n## Results\n- ABQ-LLM achieve SoTA performance in weight-only quantization\n![weight_only](fig/weight-only.png)\n- ABQ-LLM achieve SoTA performance in weight-activation quantization\n![weight_activation](fig/weight-activation.png)\n- ABQ-LLM achieve SoTA performance in zero-shot task\n![zero_shot](fig/zero-shot.png)\n- On kernel inference acceleration, ABQ- LLM achieves performance gains that far exceed those of CUTLASS and CUBLAS.\n![kernel_speed](fig/kernel_speed.png)\n- We integrated our ABQKernel into FastTransformer and compared it with the FP16 version of FastTransformer and the INT8 version of SmoothQuant. Our approach achieved a 2.8x speedup and 4.8x memory compression over FP16, using only 10GB of memory on LLaMA-30B, less than what FP16 requires for LLaMA-7B. Additionally, it outperformed SmoothQuant with a 1.6x speedup and 2.7x memory compression.\n![e2e_speed](fig/e2e_speed.png)\n\n\n## Related Project\n[SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://github.com/mit-han-lab/smoothquant)\n\n[AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://github.com/mit-han-lab/llm-awq)\n\n[GPTQ: Accurate Post-training Compression for Generative Pretrained Transformers](https://github.com/IST-DASLab/gptq)\n\n[RPTQ: Reorder-Based Post-Training Quantization for Large Language Models](https://github.com/hahnyuan/RPTQ4LLM)\n\n[OmniQuant is a simple and powerful quantization technique for LLMs](https://github.com/OpenGVLab/OmniQuant)\n\n\n## Citation\nIf you use our ABQ-LLM approach in your research, please cite our paper:\n```\n@article{zeng2024abq,\n  title={ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models},\n  author={Zeng, Chao and Liu, Songwei and Xie, Yusheng and Liu, Hong and Wang, Xiaojian and Wei, Miao and Yang, Shu and Chen, Fangmin and Mei, Xing},\n  journal={arXiv preprint arXiv:2408.08554},\n  year={2024}\n}\n```\n\n## Star History\n\n[![Star History Chart](https://api.star-history.com/svg?repos=bytedance/ABQ-LLM\u0026type=Date)](https://star-history.com/#bytedance/ABQ-LLM\u0026Date)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbytedance%2Fabq-llm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbytedance%2Fabq-llm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbytedance%2Fabq-llm/lists"}