{"id":14081277,"url":"https://github.com/huggingface/optimum-quanto","last_synced_at":"2025-05-14T04:10:32.395Z","repository":{"id":199216329,"uuid":"693745274","full_name":"huggingface/optimum-quanto","owner":"huggingface","description":"A pytorch quantization backend for optimum","archived":false,"fork":false,"pushed_at":"2025-04-24T14:55:17.000Z","size":2714,"stargazers_count":928,"open_issues_count":5,"forks_count":72,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-05-08T05:39:48.004Z","etag":null,"topics":["optimum","pytorch","quantization"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/huggingface.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-09-19T16:16:30.000Z","updated_at":"2025-05-08T01:34:31.000Z","dependencies_parsed_at":"2023-12-18T09:48:04.119Z","dependency_job_id":"dfb7fb4a-f31d-4d90-8cd4-0795ff4bb9a2","html_url":"https://github.com/huggingface/optimum-quanto","commit_stats":null,"previous_names":["huggingface/quanto","huggingface/optimum-quanto"],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Foptimum-quanto","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Foptimum-quanto/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Foptimum-quanto/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Foptimum-quanto/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/huggingface","download_url":"https://codeload.github.com/huggingface/optimum-quanto/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254070125,"owners_count":22009560,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["optimum","pytorch","quantization"],"created_at":"2024-08-13T13:00:37.957Z","updated_at":"2025-05-14T04:10:27.376Z","avatar_url":"https://github.com/huggingface.png","language":"Python","funding_links":[],"categories":["Model Storage Optimisation"],"sub_categories":[],"readme":"# Optimum Quanto\n\n🤗 Optimum Quanto is a pytorch quantization backend for [optimum](https://huggingface.co/docs/optimum/en/index).\n\nIt has been designed with versatility and simplicity in mind:\n\n- all features are available in eager mode (works with non-traceable models),\n- quantized models can be placed on any device (including CUDA and MPS),\n- automatically inserts quantization and dequantization stubs,\n- automatically inserts quantized functional operations,\n- automatically inserts quantized modules (see below the list of supported modules),\n- provides a seamless workflow from a float model to a dynamic to a static quantized model,\n- serialization compatible with pytorch `weight_only` and 🤗 `safetensors`,\n- accelerated matrix multiplications on CUDA devices (int8-int8, fp16-int4, bf16-int8, bf16-int4),\n- supports int2, int4, int8 and float8 weights,\n- supports int8 and float8 activations.\n\nFeatures yet to be implemented:\n\n- dynamic activations smoothing,\n- kernels for all mixed matrix multiplications on all devices,\n- compatibility with [torch compiler](https://pytorch.org/docs/stable/torch.compiler.html) (aka dynamo).\n\n## Performances\n\nIn a nutshell:\n\n- accuracy: models compiled with `int8`/`float8` weights and `float8` activations are very close to the full-precision models,\n- latency: whenever optimized kernels are available, the inference of quantized model is comparable with the full-precision models when quantizing only the model weights,\n- device memory: approximately divided by float bits / integer bits.\n\nThe paragraph below is just an example. Please refer to the `bench` folder for detailed results per use-case of model.\n\n### meta-llama/Meta-Llama-3.1-8B\n\n\u003cdiv class=\"row\"\u003e\u003ccenter\u003e\n  \u003cdiv class=\"column\"\u003e\n    \u003cimg src=\"https://github.com/huggingface/optimum-quanto/blob/main/bench/generation/charts/meta-llama-Meta-Llama-3.1-8B_bf16_Perplexity.png\" alt=\"meta-llama/Meta-Llama-3.1-8B WikiText perplexity\"\u003e\n  \u003c/div\u003e\n \u003c/center\u003e\n\u003c/div\u003e\n\n\u003cdiv class=\"row\"\u003e\u003ccenter\u003e\n  \u003cdiv class=\"column\"\u003e\n    \u003cimg src=\"https://github.com/huggingface/optimum-quanto/blob/main/bench/generation/charts/meta-llama-Meta-Llama-3.1-8B_bf16_Latency__ms_.png\" alt=\"meta-llama/Meta-Llama-3.1-8B Latency\"\u003e\n  \u003c/div\u003e\n \u003c/center\u003e\n\u003c/div\u003e\n\n## Installation\n\nOptimum Quanto is available as a pip package.\n\n```sh\npip install optimum-quanto\n```\n\n## Quantization workflow for Hugging Face models\n\n`optimum-quanto` provides helper classes to quantize, save and reload Hugging Face quantized models.\n\n### LLM models\n\nThe first step is to quantize the model\n\n```python\nfrom transformers import AutoModelForCausalLM\nfrom optimum.quanto import QuantizedModelForCausalLM, qint4\n\nmodel = AutoModelForCausalLM.from_pretrained('meta-llama/Meta-Llama-3-8B')\nqmodel = QuantizedModelForCausalLM.quantize(model, weights=qint4, exclude='lm_head')\n```\n\nNote: the model quantized weights will be frozen. If you want to keep them unfrozen to train them you need to use `optimum.quanto.quantize` directly.\n\nThe quantized model can be saved using `save_pretrained`:\n\n```python\nqmodel.save_pretrained('./Llama-3-8B-quantized')\n```\n\nIt can later be reloaded using `from_pretrained`:\n\n```python\nfrom optimum.quanto import QuantizedModelForCausalLM\n\nqmodel = QuantizedModelForCausalLM.from_pretrained('Llama-3-8B-quantized')\n```\n\n### Diffusers models\n\nYou can quantize any of the submodels inside a diffusers pipeline and seamlessly include them later in another pipeline.\n\nHere we quantize the `transformer` of a `Pixart` pipeline.\n\n```python\nfrom diffusers import PixArtTransformer2DModel\nfrom optimum.quanto import QuantizedPixArtTransformer2DModel, qfloat8\n\nmodel = PixArtTransformer2DModel.from_pretrained(\"PixArt-alpha/PixArt-Sigma-XL-2-1024-MS\", subfolder=\"transformer\")\nqmodel = QuantizedPixArtTransformer2DModel.quantize(model, weights=qfloat8)\nqmodel.save_pretrained(\"./pixart-sigma-fp8\")\n```\n\nLater, we can reload the quantized model and recreate the pipeline:\n\n```python\nfrom diffusers import PixArtTransformer2DModel\nfrom optimum.quanto import QuantizedPixArtTransformer2DModel\n\ntransformer = QuantizedPixArtTransformer2DModel.from_pretrained(\"./pixart-sigma-fp8\")\ntransformer.to(device=\"cuda\")\npipe = PixArtSigmaPipeline.from_pretrained(\n  \"PixArt-alpha/PixArt-Sigma-XL-2-1024-MS\",\n  transformer=None,\n  torch_dtype=torch.float16,\n).to(\"cuda\")\npipe.transformer = transformer\n```\n\n## Quantization workflow for vanilla pytorch models (low-level API)\n\nOne thing to keep in mind when using the low-level quanto API is that by default models\nweights are dynamically quantized: an explicit call must be made to 'freeze' the quantized weights.\n\nA typical quantization workflow would consist of the following steps:\n\n**1. Quantize**\n\nThe first step converts a standard float model into a dynamically quantized model.\n\n```python\nfrom optimum.quanto import quantize, qint8\n\nquantize(model, weights=qint8, activations=qint8)\n```\n\nAt this stage, only the inference of the model is modified to dynamically quantize the weights.\n\n**2. Calibrate (optional if activations are not quantized)**\n\nQuanto supports a calibration mode that allows to record the activation ranges while passing representative samples through the quantized model.\n\n```python\nfrom optimum.quanto import Calibration\n\nwith Calibration(momentum=0.9):\n    model(samples)\n```\n\nThis automatically activates the quantization of the activations in the quantized modules.\n\n\n**3. Tune, aka Quantization-Aware-Training (optional)**\n\nIf the performance of the model degrades too much, one can tune it for a few epochs to recover the float model performance.\n\n```python\nimport torch\n\nmodel.train()\nfor batch_idx, (data, target) in enumerate(train_loader):\n    data, target = data.to(device), target.to(device)\n    optimizer.zero_grad()\n    output = model(data).dequantize()\n    loss = torch.nn.functional.nll_loss(output, target)\n    loss.backward()\n    optimizer.step()\n```\n\n**4. Freeze integer weights**\n\nWhen freezing a model, its float weights are replaced by quantized integer weights.\n\n```python\nfrom optimum.quanto import freeze\n\nfreeze(model)\n```\n\n**5. Serialize quantized model**\n\nQuantized models weights can be serialized to a `state_dict`, and saved to a file.\nBoth `pickle` and `safetensors` (recommended) are supported.\n\n```python\nfrom safetensors.torch import save_file\n\nsave_file(model.state_dict(), 'model.safetensors')\n```\n\nIn order to be able to reload these weights, you also need to store the quantized\nmodel quantization map.\n\n```python\nimport json\n\nfrom optimum.quanto import quantization_map\n\nwith open('quantization_map.json', 'w') as f:\n  json.dump(quantization_map(model), f)\n```\n\n**5. Reload a quantized model**\n\nA serialized quantized model can be reloaded from a `state_dict` and a `quantization_map` using the `requantize` helper.\nNote that you need first to instantiate an empty model.\n\n```python\nimport json\n\nfrom safetensors.torch import load_file\nfrom optimum.quanto import requantize\n\nstate_dict = load_file('model.safetensors')\nwith open('quantization_map.json', 'r') as f:\n  quantization_map = json.load(f)\n\n# Create an empty model from your modeling code and requantize it\nwith torch.device('meta'):\n  new_model = ...\nrequantize(new_model, state_dict, quantization_map, device=torch.device('cuda'))\n```\n\nPlease refer to the [examples](https://github.com/huggingface/quanto/tree/main/examples) for instantiations of that workflow.\n\n\n## Design overview\n\n### Tensors\n\nAt the heart of quanto is a Tensor subclass that corresponds to:\n- the projection of a source Tensor into the optimal range for a given destination type,\n- the mapping of projected values to the destination type.\n\nFor floating-point destination types, the mapping is done by the native pytorch cast (i.e. `Tensor.to()`).\n\nFor integer destination types, the mapping is a simple rounding operation (i.e. `torch.round()`).\n\nThe goal of the projection is to increase the accuracy of the conversion by minimizing the number of:\n- saturated values (i.e. mapped to the destination type min/max),\n- zeroed values (because they are below the smallest number that can be represented by the destination type)\n\nThe projection is symmetric per-tensor or per-channel for `int8` and `float8`, and group-wise affine (with a shift or 'zero-point') for lower bitwidth.\n\nOne of the benefits of using a lower-bitwidth representation is that you will be able to take advantage of accelerated operations\nfor the destination type, which is typically faster than their higher precision equivalents.\n\nQuanto does not support the conversion of a Tensor using mixed destination types.\n\n### Modules\n\nQuanto provides a generic mechanism to replace `torch` modules by `optimum-quanto` modules that are able to process quanto tensors.\n\n`optimum-quanto` modules dynamically convert their weights until a model is frozen, which slows down inference a bit but is\nrequired if the model needs to be tuned.\n\nWeights are usually quantized per-channel along the first dimension (output features).\n\nBiases are not converted to preserve the accuracy of a typical `addmm` operation.\n\nExplanation: to be consistent with the unquantized arithmetic operations, biases would need to be quantized with a scale that\nis equal to the product of the input and weight scales, which leads to a ridiculously small scale, and conversely\nrequires a very high bitwidth to avoid clipping. Typically, with `int8` inputs and weights, biases would need to be quantized\nwith at least `12` bits, i.e. in `int16`. Since most biases are today `float16`, this is a waste of time.\n\nActivations are dynamically quantized per-tensor using static scales (defaults to the range `[-1, 1]`).\n\nTo preserve accuracy, the model needs to be calibrated to evaluate the best activation scales (using a momentum).\n\nThe following modules can be quantized:\n\n- [Linear](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html) (QLinear).\nWeights are always quantized, and biases are not quantized. Inputs and outputs can be quantized.\n- [Conv2d](https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html) (QConv2D).\nWeights are always quantized, and biases are not quantized. Inputs and outputs can be quantized.\n- [LayerNorm](https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html),\nWeights and biases are __not__ quantized. Outputs can be quantized.\n\n## Pitfalls to avoid when quantizing activations\n\nActivations are always quantized per-tensor because most linear algebra operations in a model graph are not compatible\nwith per-axis inputs: you simply cannot add numbers that are not expressed in the same base (`you cannot add apples and oranges`).\n\nWeights involved in matrix multiplications are, on the contrary, always quantized along their first axis, because all output features\nare evaluated independently from one another.\n\nThe outputs of a quantized matrix multiplication will anyway always be dequantized, even if activations are quantized, because:\n\n- the resulting accumulated values are expressed with a much higher bitwidth (typically `int32` or `float32`) than the activation bitwidth (typically `int8` or `float8`),\n- they might be combined with a `float` bias.\n\nQuantizing activations per-tensor to `int8` can lead to serious quantization errors if the corresponding tensors contain large outlier values.\nTypically, this will lead to quantized tensors with most values set to zero (except the outliers).\n\nA possible solution to work around that issue is to 'smooth' the activations statically as illustrated by [SmoothQuant](https://github.com/mit-han-lab/smoothquant).\nYou can find a script to smooth some model architectures under [external/smoothquant](external/smoothquant).\n\nA better option is to represent activations using `float8`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuggingface%2Foptimum-quanto","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhuggingface%2Foptimum-quanto","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuggingface%2Foptimum-quanto/lists"}