{"id":13409268,"url":"https://github.com/mit-han-lab/llm-awq","last_synced_at":"2025-05-14T01:09:32.431Z","repository":{"id":171786539,"uuid":"647984858","full_name":"mit-han-lab/llm-awq","owner":"mit-han-lab","description":"[MLSys 2024 Best Paper Award] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration","archived":false,"fork":false,"pushed_at":"2025-03-25T17:26:41.000Z","size":187101,"stargazers_count":2899,"open_issues_count":162,"forks_count":242,"subscribers_count":25,"default_branch":"main","last_synced_at":"2025-04-03T17:35:56.034Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mit-han-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-01T00:42:45.000Z","updated_at":"2025-04-03T09:08:57.000Z","dependencies_parsed_at":"2025-02-27T13:22:14.824Z","dependency_job_id":null,"html_url":"https://github.com/mit-han-lab/llm-awq","commit_stats":{"total_commits":103,"total_committers":13,"mean_commits":7.923076923076923,"dds":0.7669902912621359,"last_synced_commit":"a428ff86e6500f044ca581fa6ee68f3827403bca"},"previous_names":["mit-han-lab/llm-awq"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-han-lab%2Fllm-awq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-han-lab%2Fllm-awq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-han-lab%2Fllm-awq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-han-lab%2Fllm-awq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mit-han-lab","download_url":"https://codeload.github.com/mit-han-lab/llm-awq/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248294520,"owners_count":21079917,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-30T20:00:59.437Z","updated_at":"2025-04-10T20:44:43.224Z","avatar_url":"https://github.com/mit-han-lab.png","language":"Python","funding_links":[],"categories":["Models and Tools","LLM Tools","A01_文本生成_文本对话","Python","Model Storage Optimisation","Summary","🔧 **Model Compression \u0026 Optimization**","Quantization"],"sub_categories":["LLM Deployment","大语言对话模型及数据","📉 **Advanced Quantization Techniques**"],"readme":"# AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration \n[[Paper](https://arxiv.org/abs/2306.00978)][[Slides](https://www.dropbox.com/scl/fi/dtnp6h6y1mnp7g036axu6/AWQ-slide.pdf?rlkey=ffgh50hxhx8dmsnjiu8kef0ou\u0026dl=0)][[Video](https://youtu.be/3dYLj9vjfA0)]\n\n**Efficient and accurate** low-bit weight quantization (INT3/4) for LLMs, supporting **instruction-tuned** models and **multi-modal** LMs.\n\n![overview](figures/overview.png)\n\nThe current release supports: \n\n- \\[Beta\\] Chunk prefilling for faster prefilling in multi-round Q\u0026A setting. [Stable branch](https://github.com/mit-han-lab/llm-awq/tree/stable_version_20241009).\n- AWQ search for accurate quantization. \n- Pre-computed AWQ model zoo for LLMs (Llama-1/2/3, OPT, CodeLlama, StarCoder, Vicuna, VILA, LLaVA; load to generate quantized weights).\n- Memory-efficient 4-bit Linear in PyTorch.\n- Efficient CUDA kernel implementation for fast inference (support context and decoding stage).\n- Examples on 4-bit inference of an instruction-tuned model (Vicuna) and **multi-modal LM** (VILA).\n\n**Thanks to AWQ, TinyChat can deliver more efficient responses with LLM/VLM chatbots through 4-bit inference.**\n\n* TinyChat on RTX 4090 (3.4x faster than FP16):\n\n![TinyChat on RTX 4090: W4A16 is 3.4x faster than FP16](./tinychat/figures/4090_example.gif)\n\n* TinyChat on Jetson Orin (3.2x faster than FP16):\n  \n![TinyChat on Orin: W4A16 is 3.2x faster than FP16](./tinychat/figures/orin_example.gif)\n\n**TinyChat also supports inference with vision language models (e.g., VILA, LLaVA). In the following examples, W4A16 quantized models from VILA family are launched with TinyChat.**\n\n* TinyChat with VILA-13B on RTX 4090 (multi-image inputs supported):\n\n![TinyChat with VILA on 4090](./tinychat/figures/4090_vila_example.gif)\n\n* TinyChat with VILA-7B/13B on Jetson Orin:\n\n![TinyChat with VILA on Orin](./tinychat/figures/orin_vila_example.gif)\n\n\u003c!-- Check out [TinyChat](tinychat), which delievers **30 tokens/second** inference performance (**3.2x faster** than FP16) for the **Llama2** chatbot on the resource-constrained NVIDIA Jetson Orin!  --\u003e\n\nCheck out [TinyChat](tinychat), which offers a turn-key solution for **on-device inference** of LLMs and VLMs on **resource-constrained edge platforms**. With TinyChat, it is now possible to efficiently run **large** models on **small** and **low-power** devices even without Internet connection!\n\n\n## News\n- [2024/10] 🔥 \\[Beta\\] We supported **Chunk Prefilling** in TinyChat, leading to an order of magnitude faster prefilling in multi-round Q\u0026A (over 1k history tokens). Details are [here](https://github.com/mit-han-lab/llm-awq/tree/main/tinychat#new-optimization-of-context-stage). The original stable branch is [here](https://github.com/mit-han-lab/llm-awq/tree/stable_version_20241009).\n- [2024/05] 🏆 AWQ receives the **Best Paper Award** at **MLSys 2024**. 🎉 \n- [2024/05] 🔥 The **VILA-1.5** model family which features **video understanding** is now supported in AWQ and TinyChat. Check out out online demo powered by TinyChat [here](https://vila.hanlab.ai). Example is [here](scripts/vila15_example.sh).\n- [2024/05] 🔥 [AMD](https://community.amd.com/t5/ai/reduce-memory-footprint-and-improve-performance-running-llms-on/ba-p/686157) adopts AWQ to improve LLM serving efficiency.\n- [2024/04] 🔥 We released AWQ and TinyChat support for The **Llama-3** model family! Check out our example [here](scripts/llama3_example.sh).\n- [2024/02] 🔥 AWQ has been accepted to **MLSys 2024**!\n- [2024/02] 🔥 We supported [VILA Vision Languague Models](https://arxiv.org/abs/2312.07533) in AWQ \u0026 TinyChat! Check our latest demos with multi-image inputs!\n- [2024/02] 🔥 We released new version of quantized GEMM/GEMV kernels in [**TinyChat**](tinychat), leading to **38 tokens/second** inference speed on NVIDIA Jetson Orin!\n- [2024/01] 🔥 AWQ has been integrated by [Google Vertex AI](https://console.cloud.google.com/vertex-ai/publishers/meta/model-garden/llama-2-quantized)!\n- [2023/11] 🔥 AWQ has been integrated by [Amazon Sagemaker Containers](https://aws.amazon.com/blogs/machine-learning/boost-inference-performance-for-llms-with-new-amazon-sagemaker-containers/)!\n- [2023/11] 🔥 We added AWQ support and pre-computed search results for CodeLlama, StarCoder, StableCode models. Checkout our model zoo [here](https://huggingface.co/datasets/mit-han-lab/awq-model-zoo)!\n- [2023/11] 🔥 AWQ is now integrated natively in Hugging Face transformers through `from_pretrained`. You can either load quantized models from the Hub or your own HF quantized models.\n- [2023/10] AWQ is integrated into NVIDIA [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM/)\n- [2023/09] AWQ is integrated into [Intel Neural Compressor](https://github.com/intel/neural-compressor), [FastChat](https://github.com/lm-sys/FastChat/blob/main/docs/awq.md), [vLLM](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/quantization/awq.py), [HuggingFace TGI](https://github.com/huggingface/text-generation-inference/pull/1054), and [LMDeploy](https://github.com/InternLM/lmdeploy). \n- [2023/09] ⚡ Check out our latest [**TinyChat**](tinychat), which is ~2x faster than the first release on Orin!\n- [2023/09] ⚡ Check out [**AutoAWQ**](https://github.com/casper-hansen/AutoAWQ), a third-party implementation to make AWQ easier to expand to new models, improve inference speed, and integrate into Huggingface.\n- [2023/07] 🔥 We released **TinyChat**, an efficient and lightweight chatbot interface based on AWQ. TinyChat enables efficient LLM inference on both cloud and edge GPUs. Llama-2-chat models are supported! Check out our implementation [here](tinychat).\n- [2023/07] 🔥 We added AWQ support and pre-computed search results for Llama-2 models (7B \u0026 13B). Checkout our model zoo [here](https://huggingface.co/datasets/mit-han-lab/awq-model-zoo)!\n- [2023/07] We extended the support for more LLM models including MPT, Falcon, and BLOOM. \n\n## Contents\n\n- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](#awq-activation-aware-weight-quantization-for-llm-compression-and-acceleration)\n  - [News](#news)\n  - [Contents](#contents)\n  - [Install](#install)\n  - [AWQ Model Zoo](#awq-model-zoo)\n  - [Examples](#examples)\n  - [Usage](#usage)\n  - [Results on Vision-Language Models (VILA-7b/13B)](#results-on-vision-language-models-vila-7b13b)\n  - [Inference speed ( Token/sec )](#inference-speed--tokensec-)\n  - [Reference](#reference)\n  - [Related Projects](#related-projects)\n\n## Helpful Links\n\n- [VILA online demo](vila.hanlab.ai): Visual Language Models efficiently supported by AWQ \u0026 TinyChat.\n- [LLM on the Edge](https://github.com/mit-han-lab/llm-awq/tree/nv_laptop?tab=readme-ov-file#install): AWQ and TinyChat support edge GPUs such as NVIDIA Jetson Orin.\n- [VLMs on Laptop](https://github.com/mit-han-lab/llm-awq/tree/nv_laptop?tab=readme-ov-file#run-vila-on-laptop): Follow the instructions to deploy VLMs on NVIDIA Laptops with TinyChat.\n- [Gradio Server](https://github.com/mit-han-lab/llm-awq/tree/nv_laptop/tinychat/serve#gradio-demo-vila-with-tinychat): Try to build your own VLM online demo with AWQ and TinyChat!\n- [QServe](https://github.com/mit-han-lab/qserve): 🔥 **[New]** Efficient and accurate serving system for large-scale LLM inference.\n  \n## Install\n\n1. Clone this repository and navigate to AWQ folder\n```\ngit clone https://github.com/mit-han-lab/llm-awq\ncd llm-awq\n```\n\n2. Install Package\n```\nconda create -n awq python=3.10 -y\nconda activate awq\npip install --upgrade pip  # enable PEP 660 support\npip install -e .\n```\n\n* For **edge devices** like Orin, before running the commands above, please:\n\n    1. Modify [pyproject.toml](pyproject.toml) by commenting out [this line](https://github.com/mit-han-lab/llm-awq/blob/3fce69061682fdd528824e5da3d03a8a8b545f2a/pyproject.toml#L17).\n    2. Set [this line](https://github.com/mit-han-lab/llm-awq/blob/3fce69061682fdd528824e5da3d03a8a8b545f2a/pyproject.toml#18) to transformers==4.32.0.\n    3. Manually install precompiled PyTorch binaries (\u003e=2.0.0) from [NVIDIA](https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048).\n    4. Set the appropriate Python version for conda environment (e.g., `conda create -n awq python=3.8 -y` for JetPack 5).\n  \n3. Install efficient W4A16 (4-bit weight, 16-bit activation) CUDA kernel and optimized FP16 kernels (e.g. layernorm, positional encodings).\n```\ncd awq/kernels\npython setup.py install\n```\n\n4. In order to run AWQ and TinyChat with VILA-1.5 model family, please install VILA:\n\n```bash\ngit clone git@github.com:Efficient-Large-Model/VILA.git\ncd VILA\npip install -e .\n```\n\n## AWQ Model Zoo\n\nWe provide pre-computed AWQ search results for multiple model families, including LLaMA, OPT, Vicuna, and LLaVA. To get the pre-computed AWQ search results, run:\n\n```bash\n# git lfs install  # install git lfs if not already\ngit clone https://huggingface.co/datasets/mit-han-lab/awq-model-zoo awq_cache\n```\n\nThe detailed support list:\n\n| Models | Sizes                       | INT4-g128 | INT3-g128 |\n| ------ | --------------------------- | --------- | --------- |\n| [VILA-1.5](/scripts/vila15_example.sh)  | 3B/8B/13B/40B  | ✅         | ✅        |\n| [Llama3](/scripts/llama_example.sh)  | 8B/70B  | ✅         | ✅        |\n| [VILA](/scripts/vila_example.sh)    | 7B/13B                     | ✅         |           |\n| [Llama2](/scripts/llama_example.sh)  | 7B/13B/70B  | ✅         | ✅        |\n| [LLaMA](/scripts/llama2_example.sh)  | 7B/13B/30B/65B              | ✅         | ✅        |\n| [OPT](/scripts/opt_example.sh)    | 125m/1.3B/2.7B/6.7B/13B/30B | ✅         | ✅        |\n| [CodeLlama](/scripts/codellama_example.sh) | 7B/13B/34B               | ✅         | ✅        |\n| [StarCoder](/scripts/starcoder_example.sh) | 15.5B                    | ✅         | ✅        |\n| [Vicuna-v1.1](/scripts/vicuna_example.sh) | 7B/13B                 | ✅         |           |\n| [LLaVA-v0](/scripts/llava_example.sh) | 13B                       | ✅         |           |\n\nNote: We only list models that we have prepare the [AWQ searching results](https://huggingface.co/datasets/mit-han-lab/awq-model-zoo/tree/main) in the table above. AWQ also supports models such as LLaVA-v1.5 7B, and you may need to run the [AWQ search](#usage) on your own to quantize these models.\n\n## Examples\n\nAWQ can be easily applied to various LMs thanks to its good generalization, including instruction-tuned models and multi-modal LMs. It provides an easy-to-use tool to reduce the serving cost of LLMs.\n\nHere we provide two examples of AWQ application: Vicuna-7B (chatbot) and LLaVA-13B (visual reasoning) under `./examples` directory. AWQ can easily reduce the GPU memory of model serving and speed up token generation. It provides accurate quantization, providing reasoning outputs. You should be able to observe **memory savings** when running the models with 4-bit weights. \n\nNote that we perform AWQ using only textual calibration data, depsite we are running on multi-modal input. Please refer to `./examples` for details.\n\n![overview](figures/example_vis.jpg)\n\n## Usage\n\nWe provide several sample script to run AWQ (please refer to `./scripts`). We use Llama3-8B as an example.\n\n1. Perform AWQ search and save search results (we already did it for you):\n```bash\npython -m awq.entry --model_path /PATH/TO/LLAMA3/llama3-8b \\\n    --w_bit 4 --q_group_size 128 \\\n    --run_awq --dump_awq awq_cache/llama3-8b-w4-g128.pt\n```\n\n2. Evaluate the AWQ quantized model on WikiText-2 (simulated pseudo quantization)\n```bash\npython -m awq.entry --model_path /PATH/TO/LLAMA3/llama3-8b \\\n    --tasks wikitext \\\n    --w_bit 4 --q_group_size 128 \\\n    --load_awq awq_cache/llama3-8b-w4-g128.pt \\\n    --q_backend fake\n```\n\n3. Generate real quantized weights (INT4)\n```bash\nmkdir quant_cache\npython -m awq.entry --model_path /PATH/TO/LLAMA3/llama3-8b \\\n    --w_bit 4 --q_group_size 128 \\\n    --load_awq awq_cache/llama3-8b-w4-g128.pt \\\n    --q_backend real --dump_quant quant_cache/llama3-8b-w4-g128-awq.pt\n```\n\n4. Load and evaluate the real quantized model (now you can see smaller gpu memory usage)\n```bash\npython -m awq.entry --model_path /PATH/TO/LLAMA3/llama3-8b \\\n    --tasks wikitext \\\n    --w_bit 4 --q_group_size 128 \\\n    --load_quant quant_cache/llama3-8b-w4-g128-awq.pt\n```\n\n## Results on Vision-Language Models (VILA-1.5)\n\nAWQ also seamlessly supports large multi-modal models (LMMs). We demonstrate the results on the recent [VILA-1.5](https://github.com/Efficient-Large-Model/VILA) model family.\n\n\n| VILA-1.5-3B   | VQA-v2            | GQA               | VizWiz  | ScienceQA         | TextVQA           | POPE    | MME     | MMBench           | MMBench-CN    | SEED    |\n| ----------- |:-----------------:|:-----------------:|:-------:|:-----------------:|:-----------------:|:-------:|:-------:|:-----------------:|:-------------:|:-------:|\n| FP16        | 80.4  | 61.5 | 53.5   | 69.0  | 60.4  | 85.9 | 1442.4 | 63.4 | 52.7   | 60.9 |\n| AWQ-INT4    | 80.0  | 61.1 | 53.8   | 67.8  | 60.4  | 85.9 | 1437.3 | 63.3 | 51.4   | 59.8 | \n\n| VILA-1.5-8B    | VQA-v2            | GQA               | VizWiz  | ScienceQA         | TextVQA           | POPE    | MME     | MMBench           | MMBench-CN    | SEED    |\n| ----------- |:-----------------:|:-----------------:|:-------:|:-----------------:|:-----------------:|:-------:|:-------:|:-----------------:|:-------------:|:-------:|\n| FP16        | 80.9  | 61.9 | 58.7   | 79.9  | 66.3  | 84.4 | 1577.01 | 72.3 | 66.2   | 64.2 |\n| AWQ-INT4    | 80.3  | 61.7 | 59.3   | 79.0  | 65.4  | 82.9 | 1593.65 | 71.0 | 64.9   | 64.0 |\n\n| VILA-1.5-13B    | VQA-v2            | GQA               | VizWiz  | ScienceQA         | TextVQA           | POPE    | MME     | MMBench           | MMBench-CN    | SEED    |\n| ----------- |:-----------------:|:-----------------:|:-------:|:-----------------:|:-----------------:|:-------:|:-------:|:-----------------:|:-------------:|:-------:|\n| FP16       | 82.8  | 64.3 | 62.6   | 80.1  | 65.0  | 86.3 | 1569.55 | 74.9 | 66.3   | 65.1 |\n| AWQ-INT4    | 82.7  | 64.5 | 63.3   | 79.7  | 64.7  | 86.7 | 1531.35 | 74.7 | 66.7   | 65.1 |\n\n\n| VILA-1.5-40B    | VQA-v2            | GQA               | VizWiz  | ScienceQA         | TextVQA           | POPE    | MME     | MMBench           | MMBench-CN    | SEED    |\n| ----------- |:-----------------:|:-----------------:|:-------:|:-----------------:|:-----------------:|:-------:|:-------:|:-----------------:|:-------------:|:-------:|\n| FP16      | 84.3  | 64.6 | 62.2   | 87.2  | 73.6  | 87.3 | 1726.82 | 82.4 | 80.2   | 69.1 |\n| AWQ-INT4   | 84.1  | 64.4 | 61.3   | 86.7  | 73.2  | 88.2 | 1714.79 | 83.2 | 79.6   | 68.9 | \n\n\n## Inference speed ( Token/sec )\n\n| $~~~~~~$               | Precision | A100  | 4090  | Orin |\n| ---------------------- | --------- | ----- | ----- | ---- |\n| VILA1.5-3B           | fp16      | 104.6 | 137.6 | 25.4 |\n| VILA1.5-3B-AWQ       | int4      | 182.8 | 215.5 | 42.5 |\n| VILA1.5-3B-S2        | fp16      | 104.3 | 137.2 | 24.6 |\n| VILA1.5-3B-S2-AWQ    | int4      | 180.2 | 219.3 | 40.1 |\n| Llama-3-VILA1.5-8B     | fp16      | 74.9  | 57.4  | 10.2 |\n| Llama-3-VILA1.5-8B-AWQ | int4      | 168.9 | 150.2 | 28.7 |\n| VILA1.5-13B            | fp16      | 50.9  | OOM   | 6.1  |\n| VILA1.5-13B-AWQ        | int4      | 115.9 | 105.7 | 20.6 |\n| VILA1.5-40B            | fp16      | OOM  | OOM   | --  |\n| VILA1.5-40B-AWQ        | int4      | 57.0 | OOM | -- |\n\n\n## Reference\n\nIf you find AWQ useful or relevant to your research, please kindly cite our paper:\n\n```\n@inproceedings{lin2023awq,\n  title={AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration},\n  author={Lin, Ji and Tang, Jiaming and Tang, Haotian and Yang, Shang and Chen, Wei-Ming and Wang, Wei-Chen and Xiao, Guangxuan and Dang, Xingyu and Gan, Chuang and Han, Song},\n  booktitle={MLSys},\n  year={2024}\n}\n```\n\n## Related Projects\n\n[SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://github.com/mit-han-lab/smoothquant)\n\n[GPTQ: Accurate Post-training Compression for Generative Pretrained Transformers](https://arxiv.org/abs/2210.17323)\n\n[Vicuna and FastChat](https://github.com/lm-sys/FastChat#readme)\n\n[LLaVA: Large Language and Vision Assistant](https://github.com/haotian-liu/LLaVA)\n\n[VILA: On Pre-training for Visual Language Models](https://github.com/Efficient-Large-Model/VILA)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmit-han-lab%2Fllm-awq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmit-han-lab%2Fllm-awq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmit-han-lab%2Fllm-awq/lists"}