{"id":13696469,"url":"https://github.com/j-marple-dev/model_compression","last_synced_at":"2025-05-03T17:31:22.552Z","repository":{"id":47651768,"uuid":"288775532","full_name":"j-marple-dev/model_compression","owner":"j-marple-dev","description":"PyTorch Model Compression","archived":false,"fork":false,"pushed_at":"2023-01-27T01:06:14.000Z","size":32548,"stargazers_count":229,"open_issues_count":5,"forks_count":25,"subscribers_count":16,"default_branch":"master","last_synced_at":"2024-08-03T18:21:01.749Z","etag":null,"topics":["lottey-ticket-hypothesis","pruning","pytorch","quantization"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/j-marple-dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-08-19T15:58:32.000Z","updated_at":"2024-07-15T16:08:08.000Z","dependencies_parsed_at":"2023-02-15T03:35:24.519Z","dependency_job_id":null,"html_url":"https://github.com/j-marple-dev/model_compression","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j-marple-dev%2Fmodel_compression","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j-marple-dev%2Fmodel_compression/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j-marple-dev%2Fmodel_compression/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j-marple-dev%2Fmodel_compression/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/j-marple-dev","download_url":"https://codeload.github.com/j-marple-dev/model_compression/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224369562,"owners_count":17299922,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["lottey-ticket-hypothesis","pruning","pytorch","quantization"],"created_at":"2024-08-02T18:00:40.691Z","updated_at":"2024-11-13T00:30:35.164Z","avatar_url":"https://github.com/j-marple-dev.png","language":"Python","funding_links":[],"categories":["Tools","Related Repos"],"sub_categories":["Libraries","Driver"],"readme":"# Model Compression\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)\n\u003c!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section --\u003e\n[![All Contributors](https://img.shields.io/badge/all_contributors-6-orange.svg?style=flat-square)](#contributors-)\n\u003c!-- ALL-CONTRIBUTORS-BADGE:END --\u003e\n\n## Contents\n\n* [Getting started](https://github.com/j-marple-dev/model_compression#getting-started)\n  * [Prerequisites](https://github.com/j-marple-dev/model_compression#prerequisites)\n  * [Installation](https://github.com/j-marple-dev/model_compression#installation)\n  * [Docker](https://github.com/j-marple-dev/model_compression#docker)\n\n* [Usages](https://github.com/j-marple-dev/model_compression#usages)\n\n  * [Run Training](https://github.com/j-marple-dev/model_compression#run-training)\n  * [Configurations for training](https://github.com/j-marple-dev/model_compression#configurations-for-training)\n  * [Run Pruning](https://github.com/j-marple-dev/model_compression#run-pruning)\n  * [Configurations for pruning](https://github.com/j-marple-dev/model_compression#configurations-for-pruning)\n  * [Run Shrinking](https://github.com/j-marple-dev/model_compression#run-shrinking-experimental)\n  * [Run Quantization](https://github.com/j-marple-dev/model_compression#run-quantization)\n* [Experimental Results](https://github.com/j-marple-dev/model_compression#experimental-results)\n  * [Unstructured Pruning](https://github.com/j-marple-dev/model_compression#unstructured-pruning-lth-vs-weight-rewinding-vs-lr-rewinding)\n  * [Structured Pruning](https://github.com/j-marple-dev/model_compression#structured-pruning-slim-vs-l2mag-vs-l2magslim)\n  * [Shrinking after Structured Pruning](https://github.com/j-marple-dev/model_compression#shrinking-after-structured-pruning)\n  * [Quantization](https://github.com/j-marple-dev/model_compression#quantization)\n* [Class Diagram](https://github.com/j-marple-dev/model_compression#class-diagram)\n* [References](https://github.com/j-marple-dev/model_compression#references)\n* [Contributors](https://github.com/j-marple-dev/model_compression#contributors)\n\n## Getting started\n\n#### Prerequisites\n* This repository is implemented and verified on [Anaconda](https://www.anaconda.com/distribution/) virtual environment with python 3.7\n\n#### Installation\n\n0. Clone this repository.\n```bash\n$ git clone https://github.com/j-marple-dev/model_compression.git\n$ cd model_compression\n```\n\n1. Create virtual environment\n```bash\n$ conda env create -f environment.yml \n$ conda activate model_compression\n```\nor\n```bash\n$ make install \n$ conda activate model_compression\n```\n\n2. (Optional for contributors) Install CI environment\n```bash\n$ conda activate model_compression\n$ make dev\n```\n\n3. (Optional for nvidia gpu) Install cudatoolkit.\n```bash\n$ conda activate model_compression\n$ conda install -c pytorch cudatooolkit=${cuda_version}\n```\n\nAfter environment setup, you can validate the code by the following commands.\n```bash\n$ make format  # for formatting\n$ make test  # for linting\n```\n\n#### Docker\n\n0. Clone this repository.\n```bash\n$ git clone https://github.com/j-marple-dev/model_compression.git\n$ cd model_compression\n```\n\n1. Make sure you have installed [Docker Engine](https://www.docker.com/) and [nvidia-docker](https://github.com/NVIDIA/nvidia-docker).\n\n2. Run the docker image.\n```bash\n$ docker run -it --gpus all --ipc=host -v $PWD:/app/model_compression jmarpledev/model_compression:latest /bin/bash\n$ cd model_compression\n```\n\n## Usages\n\n#### Run training\nTraining the model. Trainer supports the following options:\n  - Basic Settings: batch size, epoch numbers, seed\n  - Stochastic Gradient Descent: momentum, weight decay, initial learning rate, nesterov momentum\n  - Image Augmentation: [Autoaugment](https://arxiv.org/pdf/1805.09501.pdf), [Randaugment](https://arxiv.org/pdf/1909.13719.pdf), [CutMix](https://arxiv.org/pdf/1905.04899.pdf)\n  - Loss: Cross Entropy + [Label Smoothing](https://arxiv.org/pdf/1906.02629.pdf), [Hinton Knowledge Distillation Loss](https://arxiv.org/pdf/1503.02531.pdf)\n  - Learning Rate Scheduler: [Cosine Annealing](https://arxiv.org/abs/1608.03983) with Initial Warmups\n\n```bash\n$ python train.py --help\nusage: train.py [-h] [--multi-gpu] [--gpu GPU] [--finetune FINETUNE]\n                [--resume RESUME] [--half] [--wlog] [--config CONFIG]\n\nModel trainer.\n\noptional arguments:\n  -h, --help           show this help message and exit\n  --multi-gpu          Multi-GPU use\n  --gpu GPU            GPU id to use\n  --finetune FINETUNE  Model path to finetune (.pth.tar)\n  --resume RESUME      Input log directory name to resume in save/checkpoint\n  --half               Use half precision\n  --wlog               Turns on wandb logging\n  --config CONFIG      Configuration path (.py)\n\n$ python train.py --config path_to_config.py  # basic run\n$ python train.py --config path_to_config.py  --gpu 1 --resume checkpoint_dir_name # resume training on gpu 1\n```\n\n#### Configurations for training\nFollowing options are available:\n   - Basic Settings: BATCH_SIZE, EPOCHS, SEED, MODEL_NAME(src/models), MODEL_PARAMS, DATASET\n   - Stochatic Gradient descent: MOMENTUM, WEIGHT_DECAY, LR\n   - Image Augmentation: AUG_TRAIN(src/augmentation/policies.py), AUG_TRAIN_PARAMS, AUG_TEST(src/augmentation/policies.py), CUTMIX\n   - Loss: CRITERION(src/criterions.py), CRITERION_PARAMS\n   - Learning Rate Scheduler: LR_SCHEDULER(src/lr_schedulers.py), LR_SCHEDULER_PARAMS\n\n```python\n# Example of train config(config/train/cifar/densenet_121.py)\nimport os\n\nconfig = {\n    \"SEED\": 777,\n    \"AUG_TRAIN\": \"randaugment_train_cifar100_224\",\n    \"AUG_TRAIN_PARAMS\": dict(n_select=2, level=None),\n    \"AUG_TEST\": \"simple_augment_test_cifar100_224\",\n    \"CUTMIX\": dict(beta=1, prob=0.5),\n    \"DATASET\": \"CIFAR100\",\n    \"MODEL_NAME\": \"densenet\",\n    \"MODEL_PARAMS\": dict(\n        num_classes=100,\n        inplanes=24,\n        growthRate=32,\n        compressionRate=2,\n        block_configs=(6, 12, 24, 16),\n        small_input=False,\n        efficient=False,\n    ),\n    \"CRITERION\": \"CrossEntropy\", # CrossEntropy, HintonKLD\n    \"CRITERION_PARAMS\": dict(num_classes=100, label_smoothing=0.1),\n    \"LR_SCHEDULER\": \"WarmupCosineLR\", # WarmupCosineLR, Identity, MultiStepLR\n    \"LR_SCHEDULER_PARAMS\": dict(\n        warmup_epochs=5, start_lr=1e-3, min_lr=1e-5, n_rewinding=1\n    ),\n    \"BATCH_SIZE\": 128,\n    \"LR\": 0.1,\n    \"MOMENTUM\": 0.9,\n    \"WEIGHT_DECAY\": 1e-4,\n    \"NESTEROV\": True,\n    \"EPOCHS\": 300,\n    \"N_WORKERS\": os.cpu_count(),\n}\n```\n\n\n#### Run pruning\nPruning makes a model sparse. Pruner supports the following methods:\n\n1. Unstructured Pruning\n  - [Lottery Ticket Hypothesis (LTH)](https://arxiv.org/pdf/1803.03635.pdf)\n  - [LTH with weight rewinding](https://arxiv.org/pdf/1903.01611.pdf)\n  - [LTH with learning rate rewinding](https://arxiv.org/pdf/2003.02389.pdf)\n\n2. Structured (Channel-wise) Pruning\n  - [Network Sliming](https://arxiv.org/pdf/1708.06519.pdf)\n  - [Magnitude based channel-wise pruning](http://pages.cs.wisc.edu/~kadav/app/papers/pruning.pdf)\n  - Slim-Magnitude channel-wise pruning (combination of above two methods)\n\nUsually, unstructured pruning gives more sparsity, but it doesn't support shrinking.\n\n```bash\n$ python prune.py --help\nusage: prune.py [-h] [--multi-gpu] [--gpu GPU] [--resume RESUME] [--wlog]\n                [--config CONFIG]\n\nModel pruner.\n\noptional arguments:\n  -h, --help       show this help message and exit\n  --multi-gpu      Multi-GPU use\n  --gpu GPU        GPU id to use\n  --resume RESUME  Input checkpoint directory name\n  --wlog           Turns on wandb logging\n  --config CONFIG  Configuration path\n\nusage: prune.py [-h] [--gpu GPU] [--resume RESUME] [--wlog] [--config CONFIG]\n\n$ python prune.py --config path_to_config.py  # basic run\n$ python prune.py --config path_to_config.py --multi-gpu --wlog  # run on multi-gpu with wandb logging\n```\n\n#### Configurations for pruning\nPruning configuration extends training configuration (recommended) with following options:\n  - Basic Training Settings: TRAIN_CONFIG\n  - Pruning Settings: N_PRUNING_ITER, PRUNE_METHOD(src/runner/pruner.py), PRUNE_PARAMS\n\n```python\n# Example of prune config(config/prune/cifar100/densenet_small_l2mag.py)\nfrom config.train.cifar100 import densenet_small\n\ntrain_config = densenet_small.config\nconfig = {\n    \"TRAIN_CONFIG\": train_config,\n    \"N_PRUNING_ITER\": 15,\n    \"PRUNE_METHOD\": \"Magnitude\", # LotteryTicketHypothesis, Magnitude, NetworkSlimming, SlimMagnitude\n    \"PRUNE_PARAMS\": dict(\n        PRUNE_AMOUNT=0.2,  # it iteratively prunes 20% of the network parameters at the end of trainings\n        NORM=2,\n        STORE_PARAM_BEFORE=train_config[\"EPOCHS\"],  # used for weight initialization at every pruning iteration\n        TRAIN_START_FROM=0,  # training starts from this epoch\n        PRUNE_AT_BEST=False,  # if True, it prunes parameters at the trained network which achieves the best accuracy\n                              # otherwise, it prunes the network at the end of training\n    ),\n}\n```\n\n#### Run shrinking (Experimental)\nShrinking reshapes a pruned model and reduce its size.\n\n```bash\n$ python shrink.py --help\nusage: shrink.py [-h] [--gpu GPU] [--checkpoint CHECKPOINT] [--config CONFIG]\n\nModel shrinker.\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --gpu GPU             GPU id to use\n  --checkpoint CHECKPOINT\n                        input checkpoint path to quantize\n  --config CONFIG       Pruning configuration path\n\n$ python shrink.py --config path_to_config.py --checkpoint path_to_checkpoint.pth.tar  # basic run\n```\n\n\n##### Important Notes:\nShrinker is now experimental. It only supports:\n  - channel-wise prunned models\n  - networks that consist of conv-bn-activation sequence\n  - network blocks that has channel concatenation followed by skip connections (e.g. DenseNet)\n  - networks that have only one last fully-connected layer\n\nOn the other hads, it doesn't support:\n  - network blocks that has element-wise sum followed by skip connections (e.g. ResNet, MixNet)\n  - networks that have multiple fully-connected layers\n  - Quantization after shrinking\n\n\n#### Run quantization\nIt conducts one of 8-bit quantization methods:\n  - post-training static quantization\n  - Quantization-Aware Training\n\n```bash\n$ python quantize.py --help\nusage: quantize.py [-h] [--resume RESUME] [--wlog] [--config CONFIG]\n                   [--checkpoint CHECKPOINT]\n\nModel quantizer.\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --resume RESUME       Input log directory name to resume\n  --wlog                Turns on wandb logging\n  --static              Post-training static quantization\n  --config CONFIG       Configuration path\n  --checkpoint CHECKPOINT\n                        Input checkpoint path to quantize\n\n$ python quantize.py --config path_to_config.py --checkpoint path_to_checkpoint.pth.tar  # basic qat run\n$ python quantize.py --config path_to_config.py --checkpoint path_to_checkpoint.pth.tar --static  # basic static quantization run\n```\n\n## Experiment Results\n\n[WANDB Log](https://app.wandb.ai/j-marple/model_compression/reports/Structured-Pruning-Unstructured-Pruning--VmlldzoyMTA4MDA)\n\n#### Unstructured Pruning (LTH vs Weight Rewinding vs LR Rewinding)\n\u003cimg width=\"1079\" alt=\"Screen Shot 2020-08-24 at 1 00 31 AM\" src=\"https://user-images.githubusercontent.com/14961526/90982941-962b3500-e5a5-11ea-922f-f792a2192c2e.png\"\u003e\n\n\n#### Structured Pruning (Slim vs L2Mag vs L2MagSlim)\n\u003cimg width=\"1078\" alt=\"Screen Shot 2020-08-26 at 11 05 22 PM\" src=\"https://user-images.githubusercontent.com/14961526/91313805-c6b2df00-e7f0-11ea-8e16-886c3a594247.png\"\u003e\n\n#### Shrinking after Structured Pruning\n\n###### Densenet (L=100, k=12) pruned by 19.66% (Slim \u0026 CIFAR100)\n![parameters](https://user-images.githubusercontent.com/14961526/91256808-43b76780-e7a3-11ea-965d-1543385e400a.png)\n\n* Accuracy: 80.37%\n* Parameters: 0.78M -\u003e 0.51M\n* Model Size: 6.48Mb -\u003e 4.14Mb\n\n```bash\n$ python shrink.py --config config/prune/cifar100/densenet_small_slim.py --checkpoint path_to_checkpoint.pth.tar\n\n2020-08-26 13:50:38,442 - trainer.py:71 - INFO - Created a model densenet with 0.78M params\n[Test]  100% (157 of 157) |#########################################################################################| Elapsed Time: 0:00:02 Time:  0:00:02\n2020-08-26 13:50:42,719 - shrinker.py:104 - INFO - Acc: 80.37, Size: 6.476016 MB, Sparsity: 19.66 %\n[Test]  100% (157 of 157) |#########################################################################################| Elapsed Time: 0:00:02 Time:  0:00:02\n2020-08-26 13:50:45,781 - shrinker.py:118 - INFO - Acc: 80.37, Size: 4.141713 MB, Params: 0.51 M\n```\n\n###### Densenet (L=100, k=12) pruned by 35.57% (Network Slimming \u0026 CIFAR100)\n![parameters](https://user-images.githubusercontent.com/14961526/91256890-81b48b80-e7a3-11ea-812b-d806e4afab34.png)\n\n* Accuracy: 79.07%\n* Parameters: 0.78M -\u003e 0.35M\n* Model Size: 6.48Mb -\u003e 2.85Mb\n\n```bash\n$ python shrink.py --config config/prune/cifar100/densenet_small_slim.py --checkpoint path_to_checkpoint.pth.tar\n\n2020-08-26 13:52:58,946 - trainer.py:71 - INFO - Created a model densenet with 0.78M params\n[Test]  100% (157 of 157) |#########################################################################################| Elapsed Time: 0:00:02 Time:  0:00:02\n2020-08-26 13:53:03,100 - shrinker.py:104 - INFO - Acc: 79.07, Size: 6.476016 MB, Sparsity: 35.57 %\n[Test]  100% (157 of 157) |#########################################################################################| Elapsed Time: 0:00:02 Time:  0:00:02\n2020-08-26 13:53:06,114 - shrinker.py:118 - INFO - Acc: 79.07, Size: 2.851149 MB, Params: 0.35 M\n```\n\n\n#### Quantization\n\n###### Post-training Static Quantization\n\n```bash\n$ python quantize.py --config config/quantize/cifar100/densenet_small.py --checkpoint save/test/densenet_small/296_81_20.pth.tar --static --check-acc\n\n2020-08-26 13:57:02,595 - trainer.py:71 - INFO - Created a model quant_densenet with 0.78M params\n2020-08-26 13:57:05,275 - quantizer.py:87 - INFO - Acc: 81.2 %  Size: 3.286695 MB\n2020-08-26 13:57:05,344 - quantizer.py:95 - INFO - Post Training Static Quantization: Run calibration\n[Test]  100% (157 of 157) |#########################################################################################| Elapsed Time: 0:02:40 Time:  0:02:40\n2020-08-26 13:59:47,555 - quantizer.py:117 - INFO - Acc: 81.03 %  Size: 0.974913 MB\n```\n\n###### Quantization-Aware Training\n\n```bash\n$ python quantize.py --config config/quantize/cifar100/densenet_small.py --checkpoint path_to_checkpoint.pth.tar --check-acc\n\n2020-08-26 14:06:46,855 - trainer.py:71 - INFO - Created a model quant_densenet with 0.78M params\n2020-08-26 14:06:49,506 - quantizer.py:87 - INFO - Acc: 81.2 %  Size: 3.286695 MB\n2020-08-26 14:06:49,613 - quantizer.py:99 - INFO - Quantization Aware Training: Run training\n2020-08-26 14:46:51,857 - trainer.py:209 - INFO - Epoch: [0 | 4]        train/lr: 0.0001        train/loss: 1.984219    test/loss: 1.436638     test/model_acc: 80.96%    test/best_acc: 80.96%\n[Train] 100% (782 of 782) |########################################################################################| Elapsed Time: 0:38:09 Time:  0:38:09\n[Test]  100% (157 of 157) |#########################################################################################| Elapsed Time: 0:02:40 Time:  0:02:40\n2020-08-26 15:27:43,919 - trainer.py:209 - INFO - Epoch: [1 | 4]        train/lr: 9e-05 train/loss: 1.989543    test/loss: 1.435748     test/model_acc: 80.87%    test/best_acc: 80.96%\n[Train] 100% (782 of 782) |########################################################################################| Elapsed Time: 0:38:10 Time:  0:38:10\n[Test]  100% (157 of 157) |#########################################################################################| Elapsed Time: 0:02:36 Time:  0:02:36\n2020-08-26 16:08:32,883 - trainer.py:209 - INFO - Epoch: [2 | 4]        train/lr: 6.5e-05       train/loss: 1.984149    test/loss: 1.436074     test/model_acc: 80.82%    test/best_acc: 80.96%\n[Train] 100% (782 of 782) |########################################################################################| Elapsed Time: 0:38:14 Time:  0:38:14\n[Test]  100% (157 of 157) |#########################################################################################| Elapsed Time: 0:02:39 Time:  0:02:39\n2020-08-26 16:49:28,848 - trainer.py:209 - INFO - Epoch: [3 | 4]        train/lr: 3.5e-05       train/loss: 1.984537    test/loss: 1.43442      test/model_acc: 81.01%    test/best_acc: 81.01%\n[Train] 100% (782 of 782) |########################################################################################| Elapsed Time: 0:38:19 Time:  0:38:19\n[Test]  100% (157 of 157) |#########################################################################################| Elapsed Time: 0:02:42 Time:  0:02:42\n2020-08-26 17:30:32,187 - trainer.py:209 - INFO - Epoch: [4 | 4]        train/lr: 1e-05 train/loss: 1.990936    test/loss: 1.435393     test/model_acc: 80.92%    test/best_acc: 81.01%\n[Test]  100% (157 of 157) |#########################################################################################| Elapsed Time: 0:02:37 Time:  0:02:37\n2020-08-26 17:33:10,689 - quantizer.py:117 - INFO - Acc: 81.01 %        Size: 0.974913 MB\n```\n\n## Class Diagram\n\n\u003cimg width=\"671\" alt=\"class_diagram\" src=\"https://user-images.githubusercontent.com/25141842/90982522-80684080-e5a2-11ea-812c-a2395caf9826.png\"\u003e\n\n\n## References\n\n#### Papers\n\n###### Architecture / Training\n* [MixConv: Mixed Depthwise Convolutional Kernels](https://arxiv.org/pdf/1907.09595.pdf)\n* [Densely Connected Convolutional Networks](https://arxiv.org/pdf/1608.06993.pdf)\n* [Memory-Efficient Implementation of DenseNets](https://arxiv.org/pdf/1707.06990.pdf)\n* [When Does Label Smoothing Help?](https://arxiv.org/pdf/1906.02629.pdf)\n* [SGDR: Stochastic Gradient Descent with Warm Restarts](https://arxiv.org/pdf/1608.03983.pdf)\n\n###### Augmentation\n* [Improved Regularization of Convolutional Neural Networks with Cutout](https://arxiv.org/pdf/1708.04552.pdf)\n* [AutoAugment: Learning Augmentation Strategies from Data](https://arxiv.org/pdf/1805.09501.pdf)\n* [RandAugment: Practical automated data augmentation with a reduced search space](https://arxiv.org/pdf/1909.13719.pdf)\n* [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://arxiv.org/pdf/1905.04899.pdf)\n\n###### Pruning\n* [WHAT IS THE STATE OF NEURAL NETWORK PRUNING?](https://arxiv.org/pdf/2003.03033.pdf)\n* [THE LOTTERY TICKET HYPOTHESIS: FINDING SPARSE, TRAINABLE NEURAL NETWORKS](https://arxiv.org/pdf/1803.03635.pdf)\n* [Stabilizing the Lottery Ticket Hypothesis](https://arxiv.org/pdf/1903.01611.pdf)\n* [COMPARING REWINDING AND FINE-TUNING IN NEURAL NETWORK PRUNING](https://arxiv.org/pdf/2003.02389.pdf)\n* [Learning Efficient Convolutional Networks through Network Slimming](https://arxiv.org/pdf/1708.06519.pdf)\n* [PRUNING FILTERS FOR EFFICIENT CONVNETS](http://pages.cs.wisc.edu/~kadav/app/papers/pruning.pdf)\n\n###### Knowledge Distillation\n* [The State Of Knowledge Distillation For Classification Tasks](https://arxiv.org/pdf/1912.10850.pdf)\n* [Distilling the Knowledge in a Neural Network](https://arxiv.org/pdf/1503.02531.pdf)\n\n###### Quantization\n* [Quantizing deep convolutional networks for efficient inference: A whitepaper](https://arxiv.org/pdf/1806.08342.pdf)\n\n\n#### Implementations / Tutorials\n\n###### Competition\n* https://github.com/wps712/MicroNetChallenge/tree/cifar100\n* https://github.com/Kthyeon/micronet_neurips_challenge\n\n###### Architecture / Training\n* https://github.com/rwightman/pytorch-image-models\n* https://github.com/bearpaw/pytorch-classification\n* https://github.com/gpleiss/efficient_densenet_pytorch\n* https://github.com/leaderj1001/Mixed-Depthwise-Convolutional-Kernels\n\n###### Augmentation\n* https://github.com/kakaobrain/fast-autoaugment/\n* https://github.com/DeepVoltaire/AutoAugment\n* https://github.com/clovaai/CutMix-PyTorch\n\n###### Pruning\n* https://github.com/facebookresearch/open_lth\n* https://github.com/lottery-ticket/rewinding-iclr20-public\n* https://pytorch.org/tutorials/intermediate/pruning_tutorial.html\n\n###### Knowledge Distillation\n* https://github.com/karanchahal/distiller\n\n###### Quantization\n* https://pytorch.org/docs/stable/quantization.html\n* https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html\n* https://github.com/pytorch/vision/tree/master/torchvision/models/quantization\n\n## Contributors\n\nThanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section --\u003e\n\u003c!-- prettier-ignore-start --\u003e\n\u003c!-- markdownlint-disable --\u003e\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/Curt-Park\"\u003e\u003cimg src=\"https://avatars3.githubusercontent.com/u/14961526?v=4?s=100\" width=\"100px;\" alt=\"\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eJinwoo Park (Curt)\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/j-marple-dev/model_compression/commits?author=Curt-Park\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/Hoonyyhoon\"\u003e\u003cimg src=\"https://avatars0.githubusercontent.com/u/25141842?v=4?s=100\" width=\"100px;\" alt=\"\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eJunghoon Kim\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/j-marple-dev/model_compression/commits?author=Hoonyyhoon\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/HSShin0\"\u003e\u003cimg src=\"https://avatars0.githubusercontent.com/u/44793742?v=4?s=100\" width=\"100px;\" alt=\"\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eHyungseok Shin\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/j-marple-dev/model_compression/commits?author=HSShin0\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://www.linkedin.com/in/juhee-lee-393342126/\"\u003e\u003cimg src=\"https://avatars0.githubusercontent.com/u/18753708?v=4?s=100\" width=\"100px;\" alt=\"\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eJuhee Lee\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/j-marple-dev/model_compression/commits?author=Ingenjoy\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://limjk.ai\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/10356193?v=4?s=100\" width=\"100px;\" alt=\"\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eJongkuk Lim\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/j-marple-dev/model_compression/commits?author=JeiKeiLim\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/ulken94\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/58245037?v=4?s=100\" width=\"100px;\" alt=\"\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eHaneol Kim\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/j-marple-dev/model_compression/commits?author=ulken94\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003c!-- markdownlint-restore --\u003e\n\u003c!-- prettier-ignore-end --\u003e\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:END --\u003e\n\nThis project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fj-marple-dev%2Fmodel_compression","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fj-marple-dev%2Fmodel_compression","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fj-marple-dev%2Fmodel_compression/lists"}