{"id":18264562,"url":"https://github.com/modeltc/mqbench-paper","last_synced_at":"2025-07-10T02:32:57.127Z","repository":{"id":46785211,"uuid":"373394577","full_name":"ModelTC/mqbench-paper","owner":"ModelTC","description":null,"archived":false,"fork":false,"pushed_at":"2021-07-14T02:52:05.000Z","size":6558,"stargazers_count":44,"open_issues_count":3,"forks_count":9,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-03-20T19:15:59.054Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ModelTC.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-06-03T05:45:22.000Z","updated_at":"2024-05-25T01:26:59.000Z","dependencies_parsed_at":"2022-09-06T20:50:49.613Z","dependency_job_id":null,"html_url":"https://github.com/ModelTC/mqbench-paper","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ModelTC%2Fmqbench-paper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ModelTC%2Fmqbench-paper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ModelTC%2Fmqbench-paper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ModelTC%2Fmqbench-paper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ModelTC","download_url":"https://codeload.github.com/ModelTC/mqbench-paper/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247251973,"owners_count":20908601,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-05T11:15:07.386Z","updated_at":"2025-04-04T21:30:36.550Z","avatar_url":"https://github.com/ModelTC.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MQBench: Towards Reproducible and Deployable Model Quantization Benchmark\n\n\n\nWe propose a benchmark to evaluate different quantization algorithms on various settings. MQBench is a first attempt to evaluate, analyze, and benchmark the reproducibility and deployability for model quantization algorithms. We choose multiple different platforms for real-world deployments, including CPU, GPU, ASIC, DSP, and evaluate extensive state-of-the-art quantization algorithms under a unified training pipeline. MQBench acts like a bridge to connect the algorithm and the hardware. We conduct a comprehensive analysis and find considerable intuitive or counter-intuitive insights.\n\n\n\n## Table of Contents\n\n- [Table of Contents](https://github.com/TheGreatCold/mqbench#table-of-contents)\n- [Highlighted Features](https://github.com/TheGreatCold/mqbench#highlighted-features)\n- [Installation](https://github.com/TheGreatCold/mqbench#installation)\n- [How to self-implement a quantization algorithm](https://github.com/TheGreatCold/mqbench#how-to-self-implement-a-quantization-algorithm)\n- [How to self-implement a hardware configuration](https://github.com/TheGreatCold/mqbench#how-to-self-implement-a-hardware-configuration)\n- [Submitting Your Results to MQBench](https://github.com/TheGreatCold/mqbench#submitting-your-results-to-mqbench)\n- [Lisence](https://github.com/TheGreatCold/mqbench#license)\n\n\n\n\n\n## Highlighted Features\n\n+ Integrate with the latest tracing techniques in [Pytorch](https://pytorch.org/) 1.8.\n\n+ Quantization Algorithms\n\n  + Learned Step Size Quantization: https://arxiv.org/abs/1902.08153\n  + Quantization Interval Learning: https://arxiv.org/abs/1808.05779\n  + Differentiable Soft Quantization: https://arxiv.org/abs/1908.05033\n  + Parameterized Clipping AcTivation: https://arxiv.org/abs/1805.06085\n  + Additive Powers-of-Two Quantization: https://arxiv.org/abs/1909.13144\n  + DoReFa-Net: https://arxiv.org/abs/1606.06160\n\n+ Network Architectures:\n\n  + ResNet-18, ResNet-50: https://arxiv.org/abs/1512.03385\n  + MobileNetV2: https://arxiv.org/abs/1801.04381\n  + EfficienteNet-Lite-B0: https://blog.tensorflow.org/2020/03/higher-accuracy-on-vision-models-with-efficientnet-lite.html\n  + RegNetX-600GF: https://arxiv.org/abs/2003.13678\n\n+ Hardware Platform:\n\n  | Library  | Haware Type | s Form | Granularity | Symmetry   | Fold  BN |\n  | -------- | ----------- | ------ | ----------- | ---------- | -------- |\n  | Academic | None        | FP32   | Per-tensor  | Symmetric  | No       |\n  | TensorRT | GPU         | FP32   | Per-channel | Symmetric  | Yes      |\n  | ACL      | ASIC        | FP32   | Per-channel | Asymmetric | Yes      |\n  | TVM      | ARM CPU     | POT    | Per-tensor  | Symmetric  | Yes      |\n  | SNPE     | DSP         | FP32   | Per-tensor  | Asymmetric | Yes      |\n  | FBGEMM   | X86 CPU     | FP32   | Per-channel | Asymmetric | Yes      |\n\n\n\n## Installation\n\nThese instructions will help get MQBench up.\n\n1. Clone MQBench.\n\n2. (Optionally) Create a Python virtual environment.\n\n3. Install the MQBench-required packages\n\n   `$ pip install -r requirements.txt`\n\n   Notes: MQBench uses Pytorch-1.8, our quantized model is based on the new `torch.fx` tracing techniques.\n\n4. MQBench use the Pytorch distributed data-parallel training with `nccl` backend (see details [here](https://pytorch.org/docs/stable/distributed.html#torch.distributed.init_process_group)), please make sure your machine can initailize that distributed learning environment. \n\n## How to Reproduce MQBench\n\nWe provide the running scripts `run.sh` and configuration file `config.yaml` of all experiments in MQBench. \n\nTo reproduce LSQ on ResNet-18, \n\n1. enter the directory \n\n   ```\n   $ cd PATH-TO-PROJECT/qbench_zoo\n   $ cd lsq_experiments/resnet18_4bit_academic\n   ```\n\n2. run script\n\n   ```\n   $ sh run.sh\n   ```\n\n   Note that `run.sh` contain some commands that may not be found, the core running command is\n\n   ```bash\n   PYTHONPATH=$PYTHONPATH:../../..\n   python -u -m prototype.solver.cls_quant_solver --config config.yaml\n   ```\n\n\n\n## How to self-implement a quantization algorithm\n\nAll our quantization algorithms are implemented in `prototype/quantization/`\n\nTo implementa a new algorithm, you need to add you quantizer into this directory. \n\nAll quantizer are inheritant from  `QuantizeBase` class. Each `QuantizedBase` will have an observer class which is used to estimate/update the quantization range. The observer design is inspired from the Pytorch-1.8 [repo](https://github.com/pytorch/pytorch/blob/master/torch/quantization/observer.py). Intializing a `QuantizeBase` class will also initialize a `Observer` class. \n\nThe parameters contained for  `QuantizeBase` and `Observer` include：\n\n1. `quant_min, quant_max`, which specify the $N_{min}, N_{max}$ for rounding boundaries. \n2. `qshcme`, which can be `torch.per_tensor_symmetric`, `torch.per_channel_symmetric`, `torch.per_tensor_affine`, and `torch.per_channel_affine`. This is often determined by the hardware setup. \n3. `ch_axis`, which is the dimension of channel-wise quantization. -1 is for per-tensor quantization. Typically for `nn.Conv2d` and `nn.Linear` module, the `ch_axis` should be 0.\n4. `ada_sign`, which can adaptively choose the signness. `ada_sign` should be enabled for academic setting only.\n5. `pot_scale`, which is used to determine the powers-of-two scale parameters.  \n\nNote: each specified quantizer may have its own unique parameters, see example of LSQ below.\n\n**Example Implementation of LSQ:**\n\n1. For initialization, we add new parameters for storing the scale, zero_point:\n\n   ```python\n   self.use_grad_scaling = use_grad_scaling\n   self.scale = Parameter(torch.tensor([scale]))\n   self.zero_point = Parameter(torch.tensor([zero_point]))\n   ```\n\n2. The major implementation is the `forward` function, which should contain several cases:\n\n   1. In case of `ada_sign=True`, the quantization range should be adjusted. \n\n      ```python\n      if self.ada_sign and X.min() \u003e= 0:\n        \tself.quant_max = self.activation_post_process.quant_max = 2 ** self.bitwidth - 1\n        \tself.quant_min = self.activation_post_process.quant_min = 0\n        \tself.activation_post_process.adjust_sign = True\n      ```\n\n   2. In case of symmetric quantization, the zero point should set to 0.\n\n      ```python\n      self.zero_point.data.zero_()\n      ```\n\n   3. In case of powers-of-two scale, the scale should be quantized by:\n\n      ```python\n      def pot_quantization(tensor: torch.Tensor):\n          log2t = torch.log2(tensor)\n          log2t = (torch.round(log2t)-log2t).detach() + log2t\n          return 2 ** log2t\n          \n      scale = pot_quantization(self.scale)\n      ```\n\n   4. Implement both per-channel and per-tensor quantization.\n\n**After adding you quantizer...**\n\nThe next step is to register the quantizer in `prototype/quantization/qconfig.py` \n\nImport your quantizer and then add it to `get_qconfig` function, and parse necessary arguments. \n\nThe final step is to override a `config.yaml` file:\n\n```yaml\nqparams:\n    w_method: lsq\n    a_method: lsq\n    bit: 4\n\nbackend: academic\nbnfold: 4\n```\n\nBy replacing the `w_method, a_method`, you can run your implementation. \n\nNote: the rest of the config file should not be modified in order to keep a unified training setting. \n\nHow to self-implement a hardware configuration\n-----------------------------------------------\n\nAdding a new setting in hardware is much simpler that algorithms. To do this, we can add another condition in the ``if-else`` selection. For example, adding a new hardware TFLite Micro:\n\n\n```python\n        elif backend == \"tflitemicro\":\n            backend_params = dict(ada_sign=False, symmetry=True, per_channel=False, pot_scale=True)\n        ...\n\n    model_qconfig = get_qconfig(**self.qparams, **backend_params)\n    model = quantize_fx.prepare_qat_fx(model, {\"\": model_qconfig}, foldbn_config)\n```\n\n## Submitting Your Results to MQBench\n\nYou can submit your implementation to MQBench by submmitting a merge request to this repo. The implementation of new algorithms and the running scripts, log file are needed for evalutation. \n\n\n\n## License\n\nThis project is under [Apache 2.0](https://github.com/TheGreatCold/mqbench/blob/main/LICENSE) License. \n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmodeltc%2Fmqbench-paper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmodeltc%2Fmqbench-paper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmodeltc%2Fmqbench-paper/lists"}