{"id":19731638,"url":"https://github.com/clovaai/overhaul-distillation","last_synced_at":"2025-04-06T11:08:33.013Z","repository":{"id":37664465,"uuid":"207457047","full_name":"clovaai/overhaul-distillation","owner":"clovaai","description":"Official PyTorch implementation of \"A Comprehensive Overhaul of Feature Distillation\" (ICCV 2019)","archived":false,"fork":false,"pushed_at":"2020-06-23T09:33:49.000Z","size":72,"stargazers_count":417,"open_issues_count":0,"forks_count":75,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-03-30T10:07:20.144Z","etag":null,"topics":["iccv2019","knowledge-distillation","knowledge-transfer","network-compression","teacher-student"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/clovaai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-09-10T03:26:42.000Z","updated_at":"2025-03-16T14:13:01.000Z","dependencies_parsed_at":"2022-07-12T16:42:37.618Z","dependency_job_id":null,"html_url":"https://github.com/clovaai/overhaul-distillation","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clovaai%2Foverhaul-distillation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clovaai%2Foverhaul-distillation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clovaai%2Foverhaul-distillation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clovaai%2Foverhaul-distillation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/clovaai","download_url":"https://codeload.github.com/clovaai/overhaul-distillation/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247471520,"owners_count":20944158,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["iccv2019","knowledge-distillation","knowledge-transfer","network-compression","teacher-student"],"created_at":"2024-11-12T00:22:19.854Z","updated_at":"2025-04-06T11:08:32.995Z","avatar_url":"https://github.com/clovaai.png","language":"Python","funding_links":[],"categories":["PyTorch"],"sub_categories":[],"readme":"# A Comprehensive Overhaul of Feature Distillation\n**Accepted at ICCV 2019**\n\nOfficial PyTorch implementation of \"A Comprehensive Overhaul of Feature Distillation\" | [paper](https://arxiv.org/abs/1904.01866) | [project page](https://sites.google.com/view/byeongho-heo/overhaul) | [blog](https://clova-ai.blog/2019/08/22/a-comprehensive-overhaul-of-feature-distillation-iccv-2019)\n\nByeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, Jin Young Choi\n\nClova AI Research, NAVER Corp. \\\nSeoul National University\n\n## Requirements\n- Python3\n- PyTorch (\u003e 0.4.1)\n- torchvision\n- numpy\n- scipy\n\n## Updates\n***19 Nov 2019*** Segmentation released\n\n***10 Sep 2019*** Initial upload\n\n## CIFAR-100\n\n### Settings\nWe provide the code of the experimental settings specified in the paper.\n\n| Setup | Compression type |   Teacher   |   Student   | Teacher size | Student size | Size ratio |\n|:-----:|:----------------:|:-----------:|:-----------:|:------------:|:------------:|:----------:|\n|  (a)  |       Depth      |   WRN 28-4  |   WRN 16-4  |     5.87M    |     2.77M    |    47.2%   |\n|  (b)  |      Channel     |   WRN 28-4  |   WRN 28-2  |     5.87M    |     1.47M    |    25.0%   |\n|  (c)  |  Depth \u0026 channel |   WRN 28-4  |   WRN 16-2  |     5.87M    |     0.70M    |    11.9%   |\n|  (d)  |   Architecture   |   WRN 28-4  |  ResNet 56  |     5.87M    |     0.86M    |    14.7%   |\n|  (e)  |   Architecture   | Pyramid-200 |   WRN 28-4  |    26.84M    |     5.87M    |    21.9%   |\n|  (f)  |   Architecture   | Pyramid-200 | Pyramid-110 |    26.84M    |     3.91M    |    14.6%   |\n\n### Teacher models\nDownload following pre-trained teacher network and put them into ```./data``` directory\n- [Wide Residual Network 28-4](https://drive.google.com/open?id=1Quxgs5teXVXwD3jBdkk-WeNLNpxbiZXN)\n- [PyramidNet-200(240)](https://drive.google.com/open?id=1_QgG81fNM3OvVIbMAxDPykKWuSIyKnmz)\n\n### Training\nRun ```CIFAR-100/train_with_distillation.py``` with setting alphabet (a - f)\n```\ncd CIFAR-100\npython train_with_distillation.py \\\n--setting a \\\n--epochs 200 \\\n--batch_size 128 \\\n--lr 0.1 \\\n--momentum 0.9 \\\n--weight_decay 5e-4\n```\n\nFor pyramid teacher (e, f), we used batch-size 64 to save gpu memory.\n```\ncd CIFAR-100\npython train_with_distillation.py \\\n--setting e \\\n--epochs 200 \\\n--batch_size 64 \\\n--lr 0.1 \\\n--momentum 0.9 \\\n--weight_decay 5e-4\n```\n\n### Experimental results\n\nPerformance measure is classification error rate (%)\n\n\n| Setup |   Teacher   |   Student   | Original | Proposed | Improvement |\n|:-----:|:-----------:|:-----------:|:--------:|:--------:|:-----------:|\n|  (a)  |   WRN 28-4  |   WRN 16-4  |  22.72%  |  20.89%  |    1.83%    |\n|  (b)  |   WRN 28-4  |   WRN 28-2  |  24.88%  |  21.98%  |    2.90%    |\n|  (c)  |   WRN 28-4  |   WRN 16-2  |  27.32%  |  24.08%  |    3.24%    |\n|  (d)  |   WRN 28-4  |  ResNet 56  |  27.68%  |  24.44%  |    3.24%    |\n|  (f)  | Pyramid-200 |   WRN 28-4  |  21.09%  |  17.80%  |    3.29%    |\n|  (g)  | Pyramid-200 | Pyramid-110 |  22.58%  |  18.89%  |    3.69%    |\n\n## ImageNet\n\n### Settings\n\n| Setup | Compression type |   Teacher  |  Student  | Teacher size | Student size | Size ratio |\n|:-----:|:----------------:|:----------:|:---------:|:------------:|:------------:|:----------:|\n|  (a)  |       Depth      | ResNet 152 | ResNet 50 |    60.19M    |    25.56M    |   42.47%   |\n|  (b)  |   Architecture   |  ResNet 50 | MobileNet |    25.56M    |     4.23M    |   16.55%   |\n\n\nIn case of ImageNet, teacher model will be automatically downloaded from PyTorch sites.\n\n### Training\n\n- (a) : ResNet152 to ResNet50\n```shell script\ncd ImageNet\npython train_with_distillation.py \\\n--data_path your/path/to/ImageNet \\\n--net_type resnet \\\n--epochs 100 \\\n--lr 0.1 \\\n--batch_size 256\n```\n\n- (b) : ResNet50 to MobileNet\n```shell script\ncd ImageNet\npython train_with_distillation.py \\\n--data_path your/path/to/ImageNet \\\n--net_type mobilenet \\\n--epochs 100 \\\n--lr 0.1 \\\n--batch_size 256\n```\n\n### Experimental results\n\n- ResNet 50\n\n|   Network  |  Method  | Top1-error | Top5-error |\n|:----------:|:--------:|:----------:|:----------:|\n| ResNet 152 |  Teacher |    21.69   |    5.95    |\n|  ResNet 50 | Original |    23.72   |    6.97    |\n|  ResNet 50 | Proposed |    __21.65__   |    __5.83__    |\n\n- MobileNet\n\n|  Network  |  Method  |  Top1-error |  Top5-error |\n|:---------:|:--------:|:-----:|:-----:|\n| ResNet 50 |  Teacher | 23.84 |  7.14 |\n| Mobilenet | Original | 31.13 | 11.24 |\n| Mobilenet | Proposed | __28.75__ |  __9.66__ |\n\n## Segmentation - Pascal VOC\n\nOur segmentation code is based on [pytorch-deeplab-xception](https://github.com/jfzhang95/pytorch-deeplab-xception).\n\n### Additional requirements\n\n- tqdm\n- matplotlib \n- pillow\n\n### Settings\n\n|   Teacher  |  Student  | Teacher size | Student size | Size ratio |\n|:----------:|:---------:|:------------:|:------------:|:----------:|\n| ResNet 101 | ResNet 18 |    59.3M    |    16.6    |   28.0%   |\n| ResNet 101 | MobileNetV2 |    59.3M    |     5.8M    |   9.8%   |\n\n\n### Teacher models\nDownload following pre-trained teacher network and put it into ```./Segmentation/pretrained``` directory\n- [ResNet101-DeepLabV3+](https://drive.google.com/open?id=1Pz2OT5KoSNvU5rc3w5d2R8_0OBkKSkLR)\n\nWe used pre-trained model in [pytorch-deeplab-xception](https://github.com/jfzhang95/pytorch-deeplab-xception) for teacher network.\n\n### Training\n\n- First, move to segmentation folder : ```cd Segmentation```\n- Next, configure your dataset path on ```Segmentation/mypath.py```\n\n- Without distillation\n  - ResNet 18\n  ```shell script\n  CUDA_VISIBLE_DEVICES=0,1 python train.py --backbone resnet18 --gpu-ids 0,1 --dataset pascal --use-sbd --nesterov\n  ```\n  \n  - MobileNetV2\n  ```shell script\n  CUDA_VISIBLE_DEVICES=0,1 python train.py --backbone mobilenet --gpu-ids 0,1 --dataset pascal --use-sbd --nesterov\n  ````\n\n- Distillation\n  - ResNet 18\n  ```shell script\n  CUDA_VISIBLE_DEVICES=0,1 python train_with_distillation.py --backbone resnet18 --gpu-ids 0,1 --dataset pascal --use-sbd --nesterov\n  ```\n  \n  -MobileNetV2\n  ```shell script\n  CUDA_VISIBLE_DEVICES=0,1 python train_with_distillation.py --backbone mobilenet --gpu-ids 0,1 --dataset pascal --use-sbd --nesterov\n  ```\n\n### Experimental results\n\nThis numbers are based validation performance of our code.\n\n- ResNet 18\n\n|   Network  |  Method  | mIOU |\n|:----------:|:--------:|:----------:|\n| ResNet 101 |  Teacher |    77.89   |\n| ResNet 18 | Original |    72.07   |\n| ResNet 18 | Proposed |    __73.98__   |\n\n- MobileNetV2\n\n|  Network  |  Method  |  mIOU |\n|:---------:|:--------:|:-----:|\n| ResNet 101 |  Teacher | 77.89 |\n| MobileNetV2 | Original | 68.46 |\n| MobileNetV2 | Proposed | __71.19__ |\n\n\nIn the paper, we reported performance on the **test** set, but our code measures the performance on the **val** set.\nTherefore, the performance on code is not same as the paper.\nIf you want accurate measure, please measure performance on **test** set with [Pascal VOC evaluation server](http://host.robots.ox.ac.uk/pascal/VOC/).\n\n\n## Citation\n\n```\n@inproceedings{heo2019overhaul,\n  title={A Comprehensive Overhaul of Feature Distillation},\n  author={Heo, Byeongho and Kim, Jeesoo and Yun, Sangdoo and Park, Hyojin and Kwak, Nojun and Choi, Jin Young},\n  booktitle = {International Conference on Computer Vision (ICCV)},\n  year={2019}\n}\n```\n\n## License\n\n```\nCopyright (c) 2019-present NAVER Corp.\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in\nall copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\nTHE SOFTWARE.\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclovaai%2Foverhaul-distillation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fclovaai%2Foverhaul-distillation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclovaai%2Foverhaul-distillation/lists"}