{"id":13546863,"url":"https://github.com/yoshitomo-matsubara/torchdistill","last_synced_at":"2025-05-14T01:11:28.250Z","repository":{"id":37389183,"uuid":"228891845","full_name":"yoshitomo-matsubara/torchdistill","owner":"yoshitomo-matsubara","description":"A coding-free framework built on PyTorch for reproducible deep learning studies. PyTorch Ecosystem. 🏆25 knowledge distillation methods presented at CVPR, ICLR, ECCV, NeurIPS, ICCV, etc are implemented so far. 🎁 Trained models, training logs and configurations are available for ensuring the reproducibiliy and benchmark.","archived":false,"fork":false,"pushed_at":"2025-05-08T07:11:50.000Z","size":10990,"stargazers_count":1499,"open_issues_count":0,"forks_count":133,"subscribers_count":18,"default_branch":"main","last_synced_at":"2025-05-08T08:19:55.623Z","etag":null,"topics":["amazon-sagemaker-lab","cifar10","cifar100","coco","colab-notebook","glue","google-colab","image-classification","imagenet","knowledge-distillation","natural-language-processing","nlp","object-detection","pascal-voc","pytorch","pytorch-ecosystem","semantic-segmentation","transformer"],"latest_commit_sha":null,"homepage":"https://yoshitomo-matsubara.net/torchdistill/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yoshitomo-matsubara.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.bib","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-12-18T17:40:32.000Z","updated_at":"2025-05-07T20:22:21.000Z","dependencies_parsed_at":"2024-03-02T05:22:40.758Z","dependency_job_id":"e035f39c-a977-4c91-ad6d-fd945f38b531","html_url":"https://github.com/yoshitomo-matsubara/torchdistill","commit_stats":{"total_commits":1193,"total_committers":6,"mean_commits":"198.83333333333334","dds":0.005867560771165126,"last_synced_commit":"39bf90ba514108df56d159db94d3dd99dbdc1b2f"},"previous_names":[],"tags_count":26,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yoshitomo-matsubara%2Ftorchdistill","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yoshitomo-matsubara%2Ftorchdistill/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yoshitomo-matsubara%2Ftorchdistill/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yoshitomo-matsubara%2Ftorchdistill/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yoshitomo-matsubara","download_url":"https://codeload.github.com/yoshitomo-matsubara/torchdistill/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254049432,"owners_count":22006058,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon-sagemaker-lab","cifar10","cifar100","coco","colab-notebook","glue","google-colab","image-classification","imagenet","knowledge-distillation","natural-language-processing","nlp","object-detection","pascal-voc","pytorch","pytorch-ecosystem","semantic-segmentation","transformer"],"created_at":"2024-08-01T12:00:46.507Z","updated_at":"2025-05-14T01:11:23.224Z","avatar_url":"https://github.com/yoshitomo-matsubara.png","language":"Python","funding_links":[],"categories":["🏎️ Model Compression/Acceleration","Python","Lighter and Deployment Frameworks","Computation and Communication Optimisation","PyTorch","其他_机器学习与深度学习"],"sub_categories":[],"readme":"![torchdistill logo](https://raw.githubusercontent.com/yoshitomo-matsubara/torchdistill/main/docs/source/_static/images/logo-color.png)\n\n# torchdistill: A Modular, Configuration-Driven Framework for Knowledge Distillation\n[![PyPI version](https://img.shields.io/pypi/v/torchdistill?color=00cc00)](https://pypi.org/project/torchdistill/)\n[![Build Status](https://app.travis-ci.com/yoshitomo-matsubara/torchdistill.svg?token=BsLDELKSQ5j23aFsTQKc\u0026branch=main)](https://travis-ci.com/github/yoshitomo-matsubara/torchdistill)\n[![GitHub Discussions](https://img.shields.io/github/discussions/yoshitomo-matsubara/torchdistill)](https://github.com/yoshitomo-matsubara/torchdistill/discussions)\n[![DOI:10.1007/978-3-030-76423-4_3](https://zenodo.org/badge/DOI/10.1007/978-3-030-76423-4_3.svg)](https://doi.org/10.1007/978-3-030-76423-4_3)\n[![DOI:10.18653/v1/2023.nlposs-1.18](https://zenodo.org/badge/DOI/10.18653/v1/2023.nlposs-1.18.svg)](https://doi.org/10.18653/v1/2023.nlposs-1.18)\n\n***torchdistill*** (formerly *kdkit*) offers various state-of-the-art knowledge distillation methods \nand enables you to design (new) experiments simply by editing a declarative yaml config file instead of Python code. \nEven when you need to extract intermediate representations in teacher/student models, \nyou will **NOT** need to reimplement the models, that often change the interface of the forward, but instead \nspecify the module path(s) in the yaml file. Refer to [these papers](https://github.com/yoshitomo-matsubara/torchdistill#citation) for more details.  \n\nIn addition to knowledge distillation, this framework helps you design and perform general deep learning experiments\n(**WITHOUT coding**) for reproducible deep learning studies. i.e., it enables you to train models without teachers \nsimply by excluding teacher entries from a declarative yaml config file. \nYou can find such examples below and in [configs/sample/](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/configs/sample/).   \n\nIn December 2023, ***torchdistill*** officially joined [PyTorch Ecosystem](https://pytorch.org/ecosystem/).\n\nWhen you refer to ***torchdistill*** in your paper, please cite [these papers](https://github.com/yoshitomo-matsubara/torchdistill#citation) \ninstead of this GitHub repository.  \n**If you use** ***torchdistill*** **as part of your work, your citation is appreciated and motivates me to maintain and upgrade this framework!** \n\n\n## Documentation\nYou can find the API documentation and research projects that leverage ***torchdistill*** at https://yoshitomo-matsubara.net/torchdistill/\n\n\n## Forward hook manager\nUsing **ForwardHookManager**, you can extract intermediate representations in model without modifying the interface of its forward function.  \n[This example notebook](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/demo/extract_intermediate_representations.ipynb) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/yoshitomo-matsubara/torchdistill/blob/main/demo/extract_intermediate_representations.ipynb) [![Open In Studio Lab](https://studiolab.sagemaker.aws/studiolab.svg)](https://studiolab.sagemaker.aws/import/github/yoshitomo-matsubara/torchdistill/blob/main/demo/extract_intermediate_representations.ipynb) \nwill give you a better idea of the usage such as knowledge distillation and analysis of intermediate representations.\n\nE.g., extract intermediate representations (feature map) of ResNet-18 for a random input batch\n```python\nimport torch\nfrom torchvision import models\nfrom torchdistill.core.forward_hook import ForwardHookManager\n\n# Define a model and choose torch device\nmodel = models.resnet18(pretrained=False)\ndevice = torch.device('cpu')\n\n# Register forward hooks for modules of your interest\nforward_hook_manager = ForwardHookManager(device)\nforward_hook_manager.add_hook(model, 'conv1', requires_input=True, requires_output=False)\nforward_hook_manager.add_hook(model, 'layer1.0.bn2', requires_input=True, requires_output=True)\nforward_hook_manager.add_hook(model, 'fc', requires_input=False, requires_output=True)\n\n# Define a random input batch and run the model\nx = torch.rand(32, 3, 224, 224)\ny = model(x)\n\n# Extract input and/or output of the modules\nio_dict = forward_hook_manager.pop_io_dict()\nconv1_input = io_dict['conv1']['input']\nlayer1_0_bn2_input = io_dict['layer1.0.bn2']['input']\nlayer1_0_bn2_output = io_dict['layer1.0.bn2']['output']\nfc_output = io_dict['fc']['output']\n```\n\n\n## 1 experiment → 1 declarative PyYAML config file\nIn ***torchdistill***, many components and PyTorch modules are abstracted e.g., models, datasets, optimizers, losses, \nand more! You can define them in a declarative PyYAML config file so that can be seen as a summary of your experiment, \nand in many cases, you will **NOT need to write Python code at all**. \nTake a look at some configurations available in [configs/](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/configs/). \nYou'll see what modules are abstracted and how they are defined in a declarative PyYAML config file to design an experiment.  \n\nE.g., instantiate CIFAR-10 datasets with a declarative PyYAML config file\n```python\nfrom torchdistill.common import yaml_util\nconfig = yaml_util.load_yaml_file('./test.yaml')\ntrain_dataset = config['datasets']['cifar10/train']\ntest_dataset = config['datasets']['cifar10/test']\n```\n\n**test.yaml**\n```yaml\ndatasets:\n  cifar10/train: !import_call\n    key: 'torchvision.datasets.CIFAR10'\n    init:\n      kwargs:\n        root: \u0026root_dir '~/datasets/cifar10'\n        train: True\n        download: True\n        transform: !import_call\n          key: 'torchvision.transforms.Compose'\n          init:\n            kwargs:\n              transforms:\n                - !import_call\n                  key: 'torchvision.transforms.RandomCrop'\n                  init:\n                    kwargs:\n                      size: 32\n                      padding: 4\n                - !import_call\n                  key: 'torchvision.transforms.RandomHorizontalFlip'\n                  init:\n                    kwargs:\n                      p: 0.5\n                - !import_call\n                  key: 'torchvision.transforms.ToTensor'\n                  init:\n                - !import_call\n                  key: 'torchvision.transforms.Normalize'\n                  init:\n                    kwargs: \u0026normalize_kwargs\n                      mean: [0.49139968, 0.48215841, 0.44653091]\n                      std: [0.24703223, 0.24348513, 0.26158784]\n  cifar10/test: !import_call\n    key: 'torchvision.datasets.CIFAR10'\n    init:\n      kwargs:\n        root: *root_dir\n        train: False\n        download: True\n        transform: !import_call\n          key: 'torchvision.transforms.Compose'\n          init:\n            kwargs:\n              transforms:\n                - !import_call\n                  key: 'torchvision.transforms.ToTensor'\n                  init:\n                - !import_call\n                  key: 'torchvision.transforms.Normalize'\n                  init:\n                    kwargs: *normalize_kwargs\n```\n\nIf you want to use your own modules (models, loss functions, datasets, etc) with this framework, \nyou can do so without editing code in the local package `torchdistill/`.  \nSee [the official documentation](https://yoshitomo-matsubara.net/torchdistill/usage.html) and [Discussions](https://github.com/yoshitomo-matsubara/torchdistill/discussions) for more details. \n\n## Benchmarks\n\n[Top-1 validation accuracy for ILSVRC 2012 (ImageNet)](https://yoshitomo-matsubara.net/torchdistill/benchmarks.html#imagenet-ilsvrc-2012)\n\n\n## Examples\nExecutable code can be found in [examples/](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/examples/) such as\n- [Image classification](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/examples/torchvision/image_classification.py): ImageNet (ILSVRC 2012), CIFAR-10, CIFAR-100, etc\n- [Object detection](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/examples/torchvision/object_detection.py): COCO 2017, etc\n- [Semantic segmentation](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/examples/torchvision/semantic_segmentation.py): COCO 2017, PASCAL VOC, etc\n- [Text classification](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/examples/hf_transformers/text_classification.py): GLUE, etc\n\nFor CIFAR-10 and CIFAR-100, some models are reimplemented and available as pretrained models in ***torchdistill***. \nMore details can be found [here](https://github.com/yoshitomo-matsubara/torchdistill/releases/tag/v0.1.1).  \n\nSome Transformer models fine-tuned by ***torchdistill*** for GLUE tasks are available at [Hugging Face Model Hub](https://huggingface.co/yoshitomo-matsubara). \nSample GLUE benchmark results and details can be found [here](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/examples/hf_transformers#sample-benchmark-results-and-fine-tuned-models).\n\n\n## Google Colab Examples\nThe following examples are available in [demo/](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/demo/). \nNote that these examples are for Google Colab users and compatible with Amazon SageMaker Studio Lab. \nUsually, [examples/](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/examples/) would be a better reference \nif you have your own GPU(s).\n\n### CIFAR-10 and CIFAR-100\n- Training without teacher models [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/yoshitomo-matsubara/torchdistill/blob/main/demo/cifar_training.ipynb) [![Open In Studio Lab](https://studiolab.sagemaker.aws/studiolab.svg)](https://studiolab.sagemaker.aws/import/github/yoshitomo-matsubara/torchdistill/blob/main/demo/cifar_training.ipynb)\n- Knowledge distillation [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/yoshitomo-matsubara/torchdistill/blob/main/demo/cifar_kd.ipynb) [![Open In Studio Lab](https://studiolab.sagemaker.aws/studiolab.svg)](https://studiolab.sagemaker.aws/import/github/yoshitomo-matsubara/torchdistill/blob/main/demo/cifar_kd.ipynb)\n\n### GLUE\n- Fine-tuning without teacher models [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/yoshitomo-matsubara/torchdistill/blob/main/demo/glue_finetuning_and_submission.ipynb) [![Open In Studio Lab](https://studiolab.sagemaker.aws/studiolab.svg)](https://studiolab.sagemaker.aws/import/github/yoshitomo-matsubara/torchdistill/blob/main/demo/glue_finetuning_and_submission.ipynb)\n- Knowledge distillation [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/yoshitomo-matsubara/torchdistill/blob/main/demo/glue_kd_and_submission.ipynb) [![Open In Studio Lab](https://studiolab.sagemaker.aws/studiolab.svg)](https://studiolab.sagemaker.aws/import/github/yoshitomo-matsubara/torchdistill/blob/main/demo/glue_kd_and_submission.ipynb)\n\nThese examples write out test prediction files for you to see the test performance at [the GLUE leaderboard system](https://gluebenchmark.com/).\n\n\n## PyTorch Hub\nIf you find models on [PyTorch Hub](https://pytorch.org/hub/) or GitHub repositories supporting PyTorch Hub,\nyou can import them as teacher/student models simply by editing a declarative yaml config file.  \n\ne.g., If you use a pretrained ResNeSt-50 available in [huggingface/pytorch-image-models](https://github.com/huggingface/pytorch-image-models)\n(aka *timm*) as a teacher model for ImageNet dataset, you can import the model via PyTorch Hub with the following entry \nin your declarative yaml config file.\n\n```yaml\nmodels:\n  teacher_model:\n    key: 'resnest50d'\n    repo_or_dir: 'huggingface/pytorch-image-models'\n    kwargs:\n      num_classes: 1000\n      pretrained: True\n```\n\n\n## How to setup\n- Python \u003e= 3.9\n- pipenv (optional)\n\n### Install by pip/pipenv\n```\npip3 install torchdistill\n# or use pipenv\npipenv install torchdistill\n```\n\n### Install from this repository (not recommended)\n```\ngit clone https://github.com/yoshitomo-matsubara/torchdistill.git\ncd torchdistill/\npip3 install -e .\n# or use pipenv\npipenv install \"-e .\"\n```\n\n\n## Issues / Questions / Requests / Pull Requests\nFeel free to create an issue if you find a bug.  \nIf you have either a question or feature request, start a new discussion [here](https://github.com/yoshitomo-matsubara/torchdistill/discussions).\nPlease search through [Issues](https://github.com/yoshitomo-matsubara/torchdistill/issues) and [Discussions](https://github.com/yoshitomo-matsubara/torchdistill/discussions) and make sure your issue/question/request has not been addressed yet.\n\nPull requests are welcome. \nPlease start with an issue and discuss solutions with me rather than start with a pull request. \n\n## Citation\nIf you use ***torchdistill*** in your research, please cite the following papers:  \n[[Paper](https://link.springer.com/chapter/10.1007/978-3-030-76423-4_3)] [[Preprint](https://arxiv.org/abs/2011.12913)]  \n```bibtex\n@inproceedings{matsubara2021torchdistill,\n  title={{torchdistill: A Modular, Configuration-Driven Framework for Knowledge Distillation}},\n  author={Matsubara, Yoshitomo},\n  booktitle={International Workshop on Reproducible Research in Pattern Recognition},\n  pages={24--44},\n  year={2021},\n  organization={Springer}\n}\n```\n\n[[Paper](https://aclanthology.org/2023.nlposs-1.18/)] [[OpenReview](https://openreview.net/forum?id=A5Axeeu1Bo)] [[Preprint](https://arxiv.org/abs/2310.17644)]  \n```bibtex\n@inproceedings{matsubara2023torchdistill,\n  title={{torchdistill Meets Hugging Face Libraries for Reproducible, Coding-Free Deep Learning Studies: A Case Study on NLP}},\n  author={Matsubara, Yoshitomo},\n  booktitle={Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023)},\n  publisher={Empirical Methods in Natural Language Processing},\n  pages={153--164},\n  year={2023}\n}\n```\n\n\n## Acknowledgments\n\nThis project has been supported by Travis CI's OSS credits and [JetBrain's Free License Programs (Open Source)](https://www.jetbrains.com/community/opensource/?utm_campaign=opensource\u0026utm_content=approved\u0026utm_medium=email\u0026utm_source=newsletter\u0026utm_term=jblogo#support) \nsince November 2021 and June 2022, respectively.  \n![PyCharm logo](https://resources.jetbrains.com/storage/products/company/brand/logos/PyCharm.svg)\n\n\n## References\n- [:mag:](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/examples/image_classification.py) [pytorch/vision/references/classification/](https://github.com/pytorch/vision/blob/main/references/classification/)\n- [:mag:](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/examples/object_detection.py) [pytorch/vision/references/detection/](https://github.com/pytorch/vision/tree/main/references/detection/)\n- [:mag:](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/examples/semantic_segmentation.py) [pytorch/vision/references/segmentation/](https://github.com/pytorch/vision/tree/main/references/segmentation/)\n- [:mag:](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/examples/hf_transformers/text_classification.py) [huggingface/transformers/examples/pytorch/text-classification](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification)\n- [:mag:](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/configs/sample/ilsvrc2012/kd/) Geoffrey Hinton, Oriol Vinyals, Jeff Dean. [\"Distilling the Knowledge in a Neural Network\"](https://arxiv.org/abs/1503.02531) (Deep Learning and Representation Learning Workshop: NeurIPS 2014)\n- [:mag:](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/configs/sample/ilsvrc2012/fitnet/) Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, Yoshua Bengio. [\"FitNets: Hints for Thin Deep Nets\"](https://arxiv.org/abs/1412.6550) (ICLR 2015)\n- [:mag:](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/configs/sample/ilsvrc2012/fsp/) Junho Yim, Donggyu Joo, Jihoon Bae, Junmo Kim. [\"A Gift From Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning\"](http://openaccess.thecvf.com/content_cvpr_2017/html/Yim_A_Gift_From_CVPR_2017_paper.html) (CVPR 2017)\n- [:mag:](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/configs/sample/ilsvrc2012/at/) Sergey Zagoruyko, Nikos Komodakis. [\"Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer\"](https://openreview.net/forum?id=Sks9_ajex) (ICLR 2017)\n- [:mag:](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/configs/sample/ilsvrc2012/pkt/) Nikolaos Passalis, Anastasios Tefas. [\"Learning Deep Representations with Probabilistic Knowledge Transfer\"](http://openaccess.thecvf.com/content_ECCV_2018/html/Nikolaos_Passalis_Learning_Deep_Representations_ECCV_2018_paper.html) (ECCV 2018)\n- [:mag:](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/configs/sample/ilsvrc2012/ft/) Jangho Kim, Seonguk Park, Nojun Kwak. [\"Paraphrasing Complex Network: Network Compression via Factor Transfer\"](http://papers.neurips.cc/paper/7541-paraphrasing-complex-network-network-compression-via-factor-transfer) (NeurIPS 2018)\n- [:mag:](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/configs/sample/ilsvrc2012/dab/) Byeongho Heo, Minsik Lee, Sangdoo Yun, Jin Young Choi. [\"Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons\"](https://aaai.org/ojs/index.php/AAAI/article/view/4264) (AAAI 2019)\n- [:mag:](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/configs/sample/coco2017/ktaad/) Tong He, Chunhua Shen, Zhi Tian, Dong Gong, Changming Sun, Youliang Yan. [\"Knowledge Adaptation for Efficient Semantic Segmentation\"](https://openaccess.thecvf.com/content_CVPR_2019/html/He_Knowledge_Adaptation_for_Efficient_Semantic_Segmentation_CVPR_2019_paper.html) (CVPR 2019)\n- [:mag:](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/configs/sample/ilsvrc2012/rkd/) Wonpyo Park, Dongju Kim, Yan Lu, Minsu Cho. [\"Relational Knowledge Distillation\"](http://openaccess.thecvf.com/content_CVPR_2019/html/Park_Relational_Knowledge_Distillation_CVPR_2019_paper.html) (CVPR 2019)\n- [:mag:](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/configs/sample/ilsvrc2012/vid/) Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D. Lawrence, Zhenwen Dai. [\"Variational Information Distillation for Knowledge Transfer\"](http://openaccess.thecvf.com/content_CVPR_2019/html/Ahn_Variational_Information_Distillation_for_Knowledge_Transfer_CVPR_2019_paper.html) (CVPR 2019)\n- [:mag:](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/configs/sample/ilsvrc2012/hnd/) Yoshitomo Matsubara, Sabur Baidya, Davide Callegaro, Marco Levorato, Sameer Singh. [\"Distilled Split Deep Neural Networks for Edge-Assisted Real-Time Systems\"](https://dl.acm.org/doi/10.1145/3349614.3356022) (Workshop on Hot Topics in Video Analytics and Intelligent Edges: MobiCom 2019)\n- [:mag:](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/configs/sample/ilsvrc2012/cckd/) Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, Zhaoning Zhang. [\"Correlation Congruence for Knowledge Distillation\"](http://openaccess.thecvf.com/content_ICCV_2019/html/Peng_Correlation_Congruence_for_Knowledge_Distillation_ICCV_2019_paper.html) (ICCV 2019)\n- [:mag:](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/configs/sample/ilsvrc2012/spkd/) Frederick Tung, Greg Mori. [\"Similarity-Preserving Knowledge Distillation\"](http://openaccess.thecvf.com/content_ICCV_2019/html/Tung_Similarity-Preserving_Knowledge_Distillation_ICCV_2019_paper.html) (ICCV 2019)\n- [:mag:](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/configs/sample/ilsvrc2012/crd/) Yonglong Tian, Dilip Krishnan, Phillip Isola. [\"Contrastive Representation Distillation\"](https://openreview.net/forum?id=SkgpBJrtvS) (ICLR 2020)\n- [:mag:](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/configs/sample/coco2017/ghnd/) Yoshitomo Matsubara, Marco Levorato. [\"Neural Compression and Filtering for Edge-assisted Real-time Object Detection in Challenged Networks\"](https://arxiv.org/abs/2007.15818) (ICPR 2020)\n- [:mag:](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/configs/sample/ilsvrc2012/tfkd/) Li Yuan, Francis E.H.Tay, Guilin Li, Tao Wang, Jiashi Feng. [\"Revisiting Knowledge Distillation via Label Smoothing Regularization\"](https://openaccess.thecvf.com/content_CVPR_2020/papers/Yuan_Revisiting_Knowledge_Distillation_via_Label_Smoothing_Regularization_CVPR_2020_paper.pdf) (CVPR 2020)\n- [:mag:](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/configs/sample/ilsvrc2012/sskd/) Guodong Xu, Ziwei Liu, Xiaoxiao Li, Chen Change Loy. [\"Knowledge Distillation Meets Self-Supervision\"](http://www.ecva.net/papers/eccv_2020/papers_ECCV/html/898_ECCV_2020_paper.php) (ECCV 2020)\n- [:mag:](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/configs/sample/ilsvrc2012/pad/) Youcai Zhang, Zhonghao Lan, Yuchen Dai, Fangao Zeng, Yan Bai, Jie Chang, Yichen Wei. [\"Prime-Aware Adaptive Distillation\"](http://www.ecva.net/papers/eccv_2020/papers_ECCV/html/3317_ECCV_2020_paper.php) (ECCV 2020)\n- [:mag:](https://github.com/yoshitomo-matsubara/torchdistill/tree/main/configs/sample/ilsvrc2012/kr/) Pengguang Chen, Shu Liu, Hengshuang Zhao, Jiaya Jia. [\"Distilling Knowledge via Knowledge Review\"](https://openaccess.thecvf.com/content/CVPR2021/html/Chen_Distilling_Knowledge_via_Knowledge_Review_CVPR_2021_paper.html) (CVPR 2021)\n- [:mag:](https://github.com/yoshitomo-matsubara/torchdistill/blob/main/configs/sample/ilsvrc2012/ickd/) Li Liu, Qingle Huang, Sihao Lin, Hongwei Xie, Bing Wang, Xiaojun Chang, Xiaodan Liang. [\"Exploring Inter-Channel Correlation for Diversity-Preserved Knowledge Distillation\"](https://openaccess.thecvf.com/content/ICCV2021/html/Liu_Exploring_Inter-Channel_Correlation_for_Diversity-Preserved_Knowledge_Distillation_ICCV_2021_paper.html) (ICCV 2021)\n- [:mag:](https://github.com/yoshitomo-matsubara/torchdistill/blob/main/configs/sample/ilsvrc2012/dist/) Tao Huang, Shan You, Fei Wang, Chen Qian, Chang Xu. [\"Knowledge Distillation from A Stronger Teacher\"](https://proceedings.neurips.cc/paper_files/paper/2022/hash/da669dfd3c36c93905a17ddba01eef06-Abstract-Conference.html) (NeurIPS 2022)\n- [:mag:](https://github.com/yoshitomo-matsubara/torchdistill/blob/main/configs/sample/ilsvrc2012/srd/) Roy Miles, Krystian Mikolajczyk. [\"Understanding the Role of the Projector in Knowledge Distillation\"](https://ojs.aaai.org/index.php/AAAI/article/view/28219) (AAAI 2024)\n- [:mag:](https://github.com/yoshitomo-matsubara/torchdistill/blob/main/configs/sample/ilsvrc2012/kd_w_ls/) Shangquan Sun, Wenqi Ren, Jingzhi Li, Rui Wang, Xiaochun Cao. [\"Logit Standardization in Knowledge Distillation\"](https://openaccess.thecvf.com/content/CVPR2024/html/Sun_Logit_Standardization_in_Knowledge_Distillation_CVPR_2024_paper.html) (CVPR 2024)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyoshitomo-matsubara%2Ftorchdistill","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyoshitomo-matsubara%2Ftorchdistill","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyoshitomo-matsubara%2Ftorchdistill/lists"}