{"id":15026330,"url":"https://github.com/autogptq/autogptq","last_synced_at":"2025-04-08T20:15:00.751Z","repository":{"id":153550361,"uuid":"627210550","full_name":"AutoGPTQ/AutoGPTQ","owner":"AutoGPTQ","description":"An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.","archived":false,"fork":false,"pushed_at":"2025-03-17T17:42:01.000Z","size":8399,"stargazers_count":4785,"open_issues_count":274,"forks_count":512,"subscribers_count":30,"default_branch":"main","last_synced_at":"2025-04-04T00:09:12.689Z","etag":null,"topics":["deep-learning","inference","large-language-models","llms","nlp","pytorch","quantization","transformer","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AutoGPTQ.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-13T02:18:11.000Z","updated_at":"2025-04-03T22:56:50.000Z","dependencies_parsed_at":"2024-01-12T08:47:25.115Z","dependency_job_id":"d2de4daa-e36a-4710-8f18-e7043bc59182","html_url":"https://github.com/AutoGPTQ/AutoGPTQ","commit_stats":{"total_commits":649,"total_committers":46,"mean_commits":"14.108695652173912","dds":0.5238828967642527,"last_synced_commit":"6689349625de973b9ee3016c28c11f32acf7f02c"},"previous_names":["autogptq/autogptq","panqiwei/autogptq"],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AutoGPTQ%2FAutoGPTQ","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AutoGPTQ%2FAutoGPTQ/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AutoGPTQ%2FAutoGPTQ/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AutoGPTQ%2FAutoGPTQ/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AutoGPTQ","download_url":"https://codeload.github.com/AutoGPTQ/AutoGPTQ/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247918999,"owners_count":21018045,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","inference","large-language-models","llms","nlp","pytorch","quantization","transformer","transformers"],"created_at":"2024-09-24T20:04:17.558Z","updated_at":"2025-04-08T20:15:00.737Z","avatar_url":"https://github.com/AutoGPTQ.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e 🚨 AutoGPTQ the project has reached End-of-Life 🚨 \n    \u003cbr\u003e\n🚨 Switch to \u003ca href=\"https://github.com/ModelCloud/GPTQModel\"\u003eGPTQModel\u003c/a\u003e for bug fixes and new models support 🚨\n\u003c/h1\u003e\n\n\u003ch1 align=\"center\"\u003eAutoGPTQ\u003c/h1\u003e\n\u003cp align=\"center\"\u003eAn easy-to-use LLM quantization package with user-friendly APIs, based on GPTQ algorithm (weight-only quantization).\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://github.com/PanQiWei/AutoGPTQ/releases\"\u003e\n        \u003cimg alt=\"GitHub release\" src=\"https://img.shields.io/github/release/PanQiWei/AutoGPTQ.svg\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://pypi.org/project/auto-gptq/\"\u003e\n        \u003cimg alt=\"PyPI - Downloads\" src=\"https://img.shields.io/pypi/dd/auto-gptq\"\u003e\n    \u003c/a\u003e\n\u003c/p\u003e\n\n\n## News or Update\n\n- 2024-02-15 - (News) - AutoGPTQ 0.7.0 is released, with [Marlin](https://github.com/IST-DASLab/marlin) int4*fp16 matrix multiplication kernel support, with the argument `use_marlin=True` when loading models.\n- 2023-08-23 - (News) - 🤗 Transformers, optimum and peft have integrated `auto-gptq`, so now running and training GPTQ models can be more available to everyone! See [this blog](https://huggingface.co/blog/gptq-integration) and it's resources for more details!\n\n*For more histories please turn to [here](docs/NEWS_OR_UPDATE.md)*\n\n## Performance Comparison\n\n### Inference Speed\n\u003e The result is generated using [this script](examples/benchmark/generation_speed.py), batch size of input is 1, decode strategy is beam search and enforce the model to generate 512 tokens, speed metric is tokens/s (the larger, the better).\n\u003e\n\u003e The quantized model is loaded using the setup that can gain the fastest inference speed.\n\n| model         | GPU           | num_beams | fp16  | gptq-int4 |\n|---------------|---------------|-----------|-------|-----------|\n| llama-7b      | 1xA100-40G    | 1         | 18.87 | 25.53     |\n| llama-7b      | 1xA100-40G    | 4         | 68.79 | 91.30     |\n| moss-moon 16b | 1xA100-40G    | 1         | 12.48 | 15.25     |\n| moss-moon 16b | 1xA100-40G    | 4         | OOM   | 42.67     |\n| moss-moon 16b | 2xA100-40G    | 1         | 06.83 | 06.78     |\n| moss-moon 16b | 2xA100-40G    | 4         | 13.10 | 10.80     |\n| gpt-j 6b      | 1xRTX3060-12G | 1         | OOM   | 29.55     |\n| gpt-j 6b      | 1xRTX3060-12G | 4         | OOM   | 47.36     |\n\n\n### Perplexity\nFor perplexity comparison, you can turn to [here](https://github.com/qwopqwop200/GPTQ-for-LLaMa#result) and [here](https://github.com/qwopqwop200/GPTQ-for-LLaMa#gptq-vs-bitsandbytes)\n\n## Installation\n\nAutoGPTQ is available on Linux and Windows only. You can install the latest stable release of AutoGPTQ from pip with pre-built wheels:\n\n| Platform version | Installation                                                                                      | Built against PyTorch |\n|-------------------|---------------------------------------------------------------------------------------------------|-----------------------|\n| CUDA 11.8         | `pip install auto-gptq --no-build-isolation --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/`   | 2.2.1+cu118           |\n| CUDA 12.1         | `pip install auto-gptq --no-build-isolation`                                                                            | 2.2.1+cu121           |\n| ROCm 5.7          | `pip install auto-gptq --no-build-isolation --extra-index-url https://huggingface.github.io/autogptq-index/whl/rocm573/` | 2.2.1+rocm5.7\n\nAutoGPTQ can be installed with the Triton dependency with `pip install auto-gptq[triton] --no-build-isolation` in order to be able to use the Triton backend (currently only supports linux, no 3-bits quantization).\n\nFor older AutoGPTQ, please refer to [the previous releases installation table](docs/INSTALLATION.md).\n\nOn NVIDIA systems, AutoGPTQ does not support [Maxwell or lower](https://qiita.com/uyuni/items/733a93b975b524f89f46) GPUs.\n\n### Install from source\n\nClone the source code:\n```bash\ngit clone https://github.com/PanQiWei/AutoGPTQ.git \u0026\u0026 cd AutoGPTQ\n```\n\nA few packages are required in order to build from source: `pip install numpy gekko pandas`.\n\nThen, install locally from source:\n```bash\npip install -vvv --no-build-isolation -e .\n```\nYou can set `BUILD_CUDA_EXT=0` to disable pytorch extension building, but this is **strongly discouraged** as AutoGPTQ then falls back on a slow python implementation.\n\nAs a last resort, if the above command fails, you can try `python setup.py install`.\n\n#### On ROCm systems\n\nTo install from source for AMD GPUs supporting ROCm, please specify the `ROCM_VERSION` environment variable. Example:\n\n```bash\nROCM_VERSION=5.6 pip install -vvv --no-build-isolation -e .\n```\n\nThe compilation can be speeded up by specifying the `PYTORCH_ROCM_ARCH` variable ([reference](https://github.com/pytorch/pytorch/blob/7b73b1e8a73a1777ebe8d2cd4487eb13da55b3ba/setup.py#L132)) in order to build for a single target device, for example `gfx90a` for MI200 series devices.\n\nFor ROCm systems, the packages `rocsparse-dev`, `hipsparse-dev`, `rocthrust-dev`, `rocblas-dev` and `hipblas-dev` are required to build.\n\n#### On Intel® Gaudi® 2 systems\n\n\u003eNotice: make sure you're in commit 65c2e15 or later\n\nTo install from source for Intel Gaudi 2 HPUs, set the `BUILD_CUDA_EXT=0` environment variable to disable building the CUDA PyTorch extension. Example:\n\n```bash\nBUILD_CUDA_EXT=0 pip install -vvv --no-build-isolation -e .\n```\n\n\u003eNotice that Intel Gaudi 2 uses an optimized kernel upon inference, and requires `BUILD_CUDA_EXT=0` on non-CUDA machines.\n\n## Quick Tour\n\n### Quantization and Inference\n\u003e warning: this is just a showcase of the usage of basic apis in AutoGPTQ, which uses only one sample to quantize a much small model, quality of quantized model using such little samples may not good.\n\nBelow is an example for the simplest use of `auto_gptq` to quantize a model and inference after quantization:\n```python\nfrom transformers import AutoTokenizer, TextGenerationPipeline\nfrom auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig\nimport logging\n\nlogging.basicConfig(\n    format=\"%(asctime)s %(levelname)s [%(name)s] %(message)s\", level=logging.INFO, datefmt=\"%Y-%m-%d %H:%M:%S\"\n)\n\npretrained_model_dir = \"facebook/opt-125m\"\nquantized_model_dir = \"opt-125m-4bit\"\n\ntokenizer = AutoTokenizer.from_pretrained(pretrained_model_dir, use_fast=True)\nexamples = [\n    tokenizer(\n        \"auto-gptq is an easy-to-use model quantization library with user-friendly apis, based on GPTQ algorithm.\"\n    )\n]\n\nquantize_config = BaseQuantizeConfig(\n    bits=4,  # quantize model to 4-bit\n    group_size=128,  # it is recommended to set the value to 128\n    desc_act=False,  # set to False can significantly speed up inference but the perplexity may slightly bad\n)\n\n# load un-quantized model, by default, the model will always be loaded into CPU memory\nmodel = AutoGPTQForCausalLM.from_pretrained(pretrained_model_dir, quantize_config)\n\n# quantize model, the examples should be list of dict whose keys can only be \"input_ids\" and \"attention_mask\"\nmodel.quantize(examples)\n\n# save quantized model\nmodel.save_quantized(quantized_model_dir)\n\n# save quantized model using safetensors\nmodel.save_quantized(quantized_model_dir, use_safetensors=True)\n\n# push quantized model to Hugging Face Hub.\n# to use use_auth_token=True, Login first via huggingface-cli login.\n# or pass explcit token with: use_auth_token=\"hf_xxxxxxx\"\n# (uncomment the following three lines to enable this feature)\n# repo_id = f\"YourUserName/{quantized_model_dir}\"\n# commit_message = f\"AutoGPTQ model for {pretrained_model_dir}: {quantize_config.bits}bits, gr{quantize_config.group_size}, desc_act={quantize_config.desc_act}\"\n# model.push_to_hub(repo_id, commit_message=commit_message, use_auth_token=True)\n\n# alternatively you can save and push at the same time\n# (uncomment the following three lines to enable this feature)\n# repo_id = f\"YourUserName/{quantized_model_dir}\"\n# commit_message = f\"AutoGPTQ model for {pretrained_model_dir}: {quantize_config.bits}bits, gr{quantize_config.group_size}, desc_act={quantize_config.desc_act}\"\n# model.push_to_hub(repo_id, save_dir=quantized_model_dir, use_safetensors=True, commit_message=commit_message, use_auth_token=True)\n\n# load quantized model to the first GPU\nmodel = AutoGPTQForCausalLM.from_quantized(quantized_model_dir, device=\"cuda:0\")\n\n# download quantized model from Hugging Face Hub and load to the first GPU\n# model = AutoGPTQForCausalLM.from_quantized(repo_id, device=\"cuda:0\", use_safetensors=True, use_triton=False)\n\n# inference with model.generate\nprint(tokenizer.decode(model.generate(**tokenizer(\"auto_gptq is\", return_tensors=\"pt\").to(model.device))[0]))\n\n# or you can also use pipeline\npipeline = TextGenerationPipeline(model=model, tokenizer=tokenizer)\nprint(pipeline(\"auto-gptq is\")[0][\"generated_text\"])\n```\n\nFor more advanced features of model quantization, please reference to [this script](examples/quantization/quant_with_alpaca.py)\n\n### Customize Model\n\u003cdetails\u003e\n\n\u003csummary\u003eBelow is an example to extend `auto_gptq` to support `OPT` model, as you will see, it's very easy:\u003c/summary\u003e\n\n```python\nfrom auto_gptq.modeling import BaseGPTQForCausalLM\n\n\nclass OPTGPTQForCausalLM(BaseGPTQForCausalLM):\n    # chained attribute name of transformer layer block\n    layers_block_name = \"model.decoder.layers\"\n    # chained attribute names of other nn modules that in the same level as the transformer layer block\n    outside_layer_modules = [\n        \"model.decoder.embed_tokens\", \"model.decoder.embed_positions\", \"model.decoder.project_out\",\n        \"model.decoder.project_in\", \"model.decoder.final_layer_norm\"\n    ]\n    # chained attribute names of linear layers in transformer layer module\n    # normally, there are four sub lists, for each one the modules in it can be seen as one operation,\n    # and the order should be the order when they are truly executed, in this case (and usually in most cases),\n    # they are: attention q_k_v projection, attention output projection, MLP project input, MLP project output\n    inside_layer_modules = [\n        [\"self_attn.k_proj\", \"self_attn.v_proj\", \"self_attn.q_proj\"],\n        [\"self_attn.out_proj\"],\n        [\"fc1\"],\n        [\"fc2\"]\n    ]\n```\nAfter this, you can use `OPTGPTQForCausalLM.from_pretrained` and other methods as shown in Basic.\n\n\u003c/details\u003e\n\n### Evaluation on Downstream Tasks\nYou can use tasks defined in `auto_gptq.eval_tasks` to evaluate model's performance on specific down-stream task before and after quantization.\n\nThe predefined tasks support all causal-language-models implemented in [🤗 transformers](https://github.com/huggingface/transformers) and in this project.\n\n\u003cdetails\u003e\n\n\u003csummary\u003eBelow is an example to evaluate `EleutherAI/gpt-j-6b` on sequence-classification task using `cardiffnlp/tweet_sentiment_multilingual` dataset:\u003c/summary\u003e\n\n```python\nfrom functools import partial\n\nimport datasets\nfrom transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig\n\nfrom auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig\nfrom auto_gptq.eval_tasks import SequenceClassificationTask\n\n\nMODEL = \"EleutherAI/gpt-j-6b\"\nDATASET = \"cardiffnlp/tweet_sentiment_multilingual\"\nTEMPLATE = \"Question:What's the sentiment of the given text? Choices are {labels}.\\nText: {text}\\nAnswer:\"\nID2LABEL = {\n    0: \"negative\",\n    1: \"neutral\",\n    2: \"positive\"\n}\nLABELS = list(ID2LABEL.values())\n\n\ndef ds_refactor_fn(samples):\n    text_data = samples[\"text\"]\n    label_data = samples[\"label\"]\n\n    new_samples = {\"prompt\": [], \"label\": []}\n    for text, label in zip(text_data, label_data):\n        prompt = TEMPLATE.format(labels=LABELS, text=text)\n        new_samples[\"prompt\"].append(prompt)\n        new_samples[\"label\"].append(ID2LABEL[label])\n\n    return new_samples\n\n\n#  model = AutoModelForCausalLM.from_pretrained(MODEL).eval().half().to(\"cuda:0\")\nmodel = AutoGPTQForCausalLM.from_pretrained(MODEL, BaseQuantizeConfig())\ntokenizer = AutoTokenizer.from_pretrained(MODEL)\n\ntask = SequenceClassificationTask(\n        model=model,\n        tokenizer=tokenizer,\n        classes=LABELS,\n        data_name_or_path=DATASET,\n        prompt_col_name=\"prompt\",\n        label_col_name=\"label\",\n        **{\n            \"num_samples\": 1000,  # how many samples will be sampled to evaluation\n            \"sample_max_len\": 1024,  # max tokens for each sample\n            \"block_max_len\": 2048,  # max tokens for each data block\n            # function to load dataset, one must only accept data_name_or_path as input\n            # and return datasets.Dataset\n            \"load_fn\": partial(datasets.load_dataset, name=\"english\"),\n            # function to preprocess dataset, which is used for datasets.Dataset.map,\n            # must return Dict[str, list] with only two keys: [prompt_col_name, label_col_name]\n            \"preprocess_fn\": ds_refactor_fn,\n            # truncate label when sample's length exceed sample_max_len\n            \"truncate_prompt\": False\n        }\n    )\n\n# note that max_new_tokens will be automatically specified internally based on given classes\nprint(task.run())\n\n# self-consistency\nprint(\n    task.run(\n        generation_config=GenerationConfig(\n            num_beams=3,\n            num_return_sequences=3,\n            do_sample=True\n        )\n    )\n)\n```\n\n\u003c/details\u003e\n\n## Learn More\n[tutorials](docs/tutorial) provide step-by-step guidance to integrate `auto_gptq` with your own project and some best practice principles.\n\n[examples](examples/README.md) provide plenty of example scripts to use `auto_gptq` in different ways.\n\n## Supported Models\n\n\u003e you can use `model.config.model_type` to compare with the table below to check whether the model you use is supported by `auto_gptq`.\n\u003e\n\u003e for example, model_type of `WizardLM`, `vicuna` and `gpt4all` are all `llama`, hence they are all supported by `auto_gptq`.\n\n| model type                         | quantization | inference | peft-lora | peft-ada-lora | peft-adaption_prompt                                                                            |\n|------------------------------------|--------------|-----------|-----------|---------------|-------------------------------------------------------------------------------------------------|\n| bloom                              | ✅            | ✅         | ✅         | ✅             |                                                                                                 |\n| gpt2                               | ✅            | ✅         | ✅         | ✅             |                                                                                                 |\n| gpt_neox                           | ✅            | ✅         | ✅         | ✅             | ✅[requires this peft branch](https://github.com/PanQiWei/peft/tree/multi_modal_adaption_prompt) |\n| gptj                               | ✅            | ✅         | ✅         | ✅             | ✅[requires this peft branch](https://github.com/PanQiWei/peft/tree/multi_modal_adaption_prompt) |\n| llama                              | ✅            | ✅         | ✅         | ✅             | ✅                                                                                               |\n| moss                               | ✅            | ✅         | ✅         | ✅             | ✅[requires this peft branch](https://github.com/PanQiWei/peft/tree/multi_modal_adaption_prompt) |\n| opt                                | ✅            | ✅         | ✅         | ✅             |                                                                                                 |\n| gpt_bigcode                        | ✅            | ✅         | ✅         | ✅             |                                                                                                 |\n| codegen                            | ✅            | ✅         | ✅         | ✅             |                                                                                                 |\n| falcon(RefinedWebModel/RefinedWeb) | ✅            | ✅         | ✅         | ✅             |                                                                                                 |\n\n## Supported Evaluation Tasks\nCurrently, `auto_gptq` supports: `LanguageModelingTask`, `SequenceClassificationTask` and `TextSummarizationTask`; more Tasks will come soon!\n\n## Running tests\n\nTests can be run with:\n\n```\npytest tests/ -s\n```\n\n## FAQ\n\n### Which kernel is used by default?\n\nAutoGPTQ defaults to using exllamav2 int4*fp16 kernel for matrix multiplication.\n\n### How to use Marlin kernel?\n\nMarlin is an optimized int4 * fp16 kernel was recently proposed at https://github.com/IST-DASLab/marlin. This is integrated in AutoGPTQ when loading a model with `use_marlin=True`. This kernel is available only on devices with compute capability 8.0 or 8.6 (Ampere GPUs).\n\n## Acknowledgement\n- Special thanks **Elias Frantar**, **Saleh Ashkboos**, **Torsten Hoefler** and **Dan Alistarh** for proposing **GPTQ** algorithm and open source the [code](https://github.com/IST-DASLab/gptq), and for releasing [Marlin kernel](https://github.com/IST-DASLab/marlin) for mixed precision computation.\n- Special thanks **qwopqwop200**, for code in this project that relevant to quantization are mainly referenced from [GPTQ-for-LLaMa](https://github.com/qwopqwop200/GPTQ-for-LLaMa/tree/cuda).\n- Special thanks to **turboderp**, for releasing [Exllama](https://github.com/turboderp/exllama) and [Exllama v2](https://github.com/turboderp/exllamav2) libraries with efficient mixed precision kernels.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautogptq%2Fautogptq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fautogptq%2Fautogptq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautogptq%2Fautogptq/lists"}