{"id":13443252,"url":"https://github.com/facebookresearch/kill-the-bits","last_synced_at":"2025-03-20T16:30:58.664Z","repository":{"id":56440277,"uuid":"187827243","full_name":"facebookresearch/kill-the-bits","owner":"facebookresearch","description":"Code for: \"And the bit goes down: Revisiting the quantization of neural networks\"","archived":true,"fork":false,"pushed_at":"2020-11-09T09:57:19.000Z","size":26868,"stargazers_count":633,"open_issues_count":0,"forks_count":123,"subscribers_count":25,"default_branch":"master","last_synced_at":"2024-12-17T01:37:37.559Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/facebookresearch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-05-21T11:52:40.000Z","updated_at":"2024-11-24T02:43:33.000Z","dependencies_parsed_at":"2022-08-15T18:50:14.034Z","dependency_job_id":null,"html_url":"https://github.com/facebookresearch/kill-the-bits","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fkill-the-bits","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fkill-the-bits/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fkill-the-bits/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fkill-the-bits/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/facebookresearch","download_url":"https://codeload.github.com/facebookresearch/kill-the-bits/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244649778,"owners_count":20487489,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T03:01:58.149Z","updated_at":"2025-03-20T16:30:58.659Z","avatar_url":"https://github.com/facebookresearch.png","language":"Python","funding_links":[],"categories":["Python","Tools"],"sub_categories":["Paper Implementations"],"readme":"# And the bit goes down\n\nThis repository contains the implementation of our paper: [And the bit goes down: Revisiting the quantization of neural networks](https://arxiv.org/abs/1907.05686) (ICLR 2020) as well as the compressed models we obtain (ResNets and Mask R-CNN).\n\nOur compression method is based on vector quantization. It takes as input an already trained neural network and, through a distillation procedure at all layers and a fine-tuning stage, optimizes the accuracy of the network.\n\nThis  approach  outperforms  the  state-of-the-art  w.r.t.  compression/accuracy  trade-off for standard networks like ResNet-18 and ResNet-50 (see  [Compressed models](#Compressed-Models)).\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"illustration.png\" alt=\"Illustration of our method. We approximate a binary classifier $\\varphi$ that labels images $x$ as dogs or cats by quantizing its weights.\n    \\textbf{Standard method}: quantizing $\\varphi$ with the standard objective function \\eqref{eq:pq_obj} leads to a classifier $\\widehat \\varphi_{\\text{bad}}$ that tries to approximate $\\varphi$ over the entire input space and thus performs badly for in-domains inputs.\n    \\textbf{Our method}: quantizing $\\varphi$ with our objective function \\eqref{eq:ours_obj} leads to a classifier  $\\widehat \\varphi_{\\text{good}}$ that performs well for in-domain inputs.\n    \\benjamin{Maybe move the $\\widehat \\varphi_{\\text{bad}}$ and $\\widehat \\varphi_{\\text{good}}$ labels from the top left corner into the image close to the respective lines. Then the $\\phi(x)$ can come down a little.\"\"\u003e\n\u003c/p\u003e\n\n\n## Installation\n\nOur code works with Python 3.6 and newest. To run the code, you must have the following packages installed:\n- [NumPy](http://www.numpy.org/)\n- [PyTorch](http://pytorch.org/) (version=1.0.1.post2)\n\nThese dependencies can be installed with:\n`\npip install -r requirements.txt\n`\n\n## Compressed Models\nThe compressed models (centroids + assignments) are available in the `models/compressed` folder. We provide code to evaluate those models on their standard benchmarks (ImageNet/COCO). Note that inference can be performed both on GPU or on CPU. Note also that we did not optimize this precise part of the code for speed. Indeed, the code for inference should rather be regarded as a proof of concept: based on the centroids and the assignments, we recover the accuracies mentioned in the table above by instantiating the full, non-compressed model.\n\n### Vanilla ResNets\nWe provide the vanilla compressed ResNet-18 and ResNet-50 models for 256 centroids in the low and high compression regimes. As mentioned in the paper, the low compression regime corresponds to a block size of 9 for standard 3x3 convolutions and to a block size of 4 for 1x1 pointwise convolutions. Similarly, the high compression regime corresponds to a block size of 18 for standard 3x3 convolutions and to a block size of 8 for 1x1 pointwise convolutions.\n\n|Model (non-compressed top-1) | Compression | Size ratio | Model size | Top-1 (%)|   \n|:-:|:-:|:-:|:-:|:--:|\nResNet-18 (69.76%) | Small blocks \u003cbr\u003eLarge blocks | 29x \u003cbr\u003e43x |1.54 MB\u003cbr\u003e1.03 MB|**65.81**\u003cbr\u003e**61.18**\nResNet-50 (76.15%) | Small blocks \u003cbr\u003eLarge blocks | 19x \u003cbr\u003e31x |5.09 MB\u003cbr\u003e3.19 MB|**73.79**\u003cbr\u003e**68.21**\n\nTo evaluate on the standard test set of ImageNet: clone the repo, `cd` into `src/` and run:\n```bash\npython inference.py --model resnet18 --state-dict-compressed models/compressed/resnet18_small_blocks.pth --device cuda --data-path YOUR_IMAGENET_PATH\n```\n\n### Semi-supervised ResNet50\nWe provide the compressed [semi-supervised ResNet50](https://arxiv.org/abs/1905.00546) trained and open-sourced by Yalniz *et. al.* We use 256 centroids and the small blocks compression regime.\n\n\n|Model (non-compressed top-1) | Compression | Size ratio | Model size | Top-1 (%)|   \n|:-:|:-:|:-:|:-:|:--:|\nSemi-Supervised ResNet-50 (79.30%) | Small blocks| 19x | 5.20 MB | **76.12**\n\nTo evaluate on the standard test set of ImageNet: clone the repo, `cd` into `src/` and run:\n```bash\npython inference.py --model resnet50_semisup --state-dict-compressed models/compressed/resnet50_semisup_small_blocks.pth --device cuda --data-path YOUR_IMAGENET_PATH\n```\n\n### Mask R-CNN\n\nWe provide the compressed Mask R-CNN (backbone ResNet50-FPN) available in the [PyTorch Model Zoo](https://pytorch.org/docs/stable/torchvision/models.html). As mentioned in the paper, we use 256 centroids and various block sizes to reach an interesting size/accuracy tradeoff (with a 26x compression factor). Note that you need [torchvision 0.3](https://pytorch.org/blog/torchvision03/) in order to run this part of the code.\n\n|Model | Size | Box AP| Mask AP |   \n|:-:|:-:|:-:|:-:|\n|Non-compressed | 170 MB | 37.9 | 34.6|\n|Compressed | 6.65 MB | 33.9 | 30.8 |\n\nTo evaluate on COCO: clone the repo, run `git checkout mask_r_cnn`, `cd` into `src/` and run:\n```bash\npython inference.py --model maskrcnn_resnet50_fpn --state-dict-compressed models/compressed/mask_r_cnn.pth --device cuda --data-path YOUR_COCO_PATH\n```\n\n## Results\n\nYou can also compress the vanilla ResNet models and reproduce the results of our paper by `cd` into `src/` and by running the following commands:\n- For the *small blocks* compression regime:\n```bash\npython quantize.py --model resnet18 --block-size-cv 9 --block-size-pw 4 --n-centroids-cv 256 --n-centroids-pw 256 --n-centroids-fc 2048 --data-path YOUR_IMAGENET_PATH\npython quantize.py --model resnet50 --block-size-cv 9 --block-size-pw 4 --n-centroids-cv 256 --n-centroids-pw 256 --n-centroids-fc 1024 --data-path YOUR_IMAGENET_PATH\n```\n- For the *large blocks* compression regime:\n```bash\npython quantize.py --model resnet18 --block-size-cv 18 --block-size-pw 4 --n-centroids-cv 256 --n-centroids-pw 256 --n-centroids-fc 2048 --data-path YOUR_IMAGENET_PATH\npython quantize.py --model resnet50 --block-size-cv 18 --block-size-pw 8 --n-centroids-cv 256 --n-centroids-pw 256 --n-centroids-fc 1024 --data-path YOUR_IMAGENET_PATH\n```\nNote that the vanilla ResNet-18 and ResNet-50 teacher (non-compressed) models are taken from the PyTorch model zoo. Note also that we run our code on a single 16GB Volta V100 GPU.\n\n## License\nThis repository is released under Creative Commons Attribution 4.0 International (CC BY 4.0) license, as found in the LICENSE file.\n\n## Bibliography\nPlease consider citing [1] if you found the resources in this repository useful.\n\n[1] Stock, Pierre and Joulin, Armand and Gribonval, Rémi and Graham, Benjamin and Jégou, Hervé. [And the bit goes down: Revisiting the quantization of neural networks](https://arxiv.org/abs/1907.05686).\n```\n@inproceedings{stock2019killthebits,\n  title = {And the bit goes down: Revisiting the quantization of neural networks},\n  author = {Stock, Pierre and Joulin, Armand and Gribonval, R{\\'e}mi and Graham, Benjamin and J{\\'e}gou, Herv{\\'e}},\n  booktitle = {International Conference on Learning Representations (ICLR)},\n  year = {2020}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffacebookresearch%2Fkill-the-bits","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffacebookresearch%2Fkill-the-bits","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffacebookresearch%2Fkill-the-bits/lists"}