{"id":13486286,"url":"https://github.com/SwinTransformer/Feature-Distillation","last_synced_at":"2025-03-27T20:32:43.487Z","repository":{"id":44426854,"uuid":"496909645","full_name":"SwinTransformer/Feature-Distillation","owner":"SwinTransformer","description":null,"archived":false,"fork":false,"pushed_at":"2022-11-30T08:49:53.000Z","size":179,"stargazers_count":253,"open_issues_count":18,"forks_count":11,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-03-24T11:54:52.425Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SwinTransformer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-05-27T07:57:57.000Z","updated_at":"2025-03-19T11:03:35.000Z","dependencies_parsed_at":"2023-01-21T20:02:34.831Z","dependency_job_id":null,"html_url":"https://github.com/SwinTransformer/Feature-Distillation","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SwinTransformer%2FFeature-Distillation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SwinTransformer%2FFeature-Distillation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SwinTransformer%2FFeature-Distillation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SwinTransformer%2FFeature-Distillation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SwinTransformer","download_url":"https://codeload.github.com/SwinTransformer/Feature-Distillation/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245920405,"owners_count":20694082,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T18:00:43.226Z","updated_at":"2025-03-27T20:32:43.457Z","avatar_url":"https://github.com/SwinTransformer.png","language":"Python","funding_links":[],"categories":["Uncategorized","Fundamental MIM Methods"],"sub_categories":["Uncategorized","MIM for Transformers"],"readme":"# Feature-Distillation\n\nBy [Yixuan Wei](https://scholar.google.com/citations?user=xwudKb4AAAAJ\u0026hl=en)\\*, [Han Hu](https://ancientmooner.github.io/)\\*, [Zhenda Xie](https://zdaxie.github.io), [Zheng Zhang](https://stupidzz.github.io/), [Yue Cao](http://yue-cao.me), [Jianmin Bao](https://jianminbao.github.io/), [Dong Chen](http://www.dongchen.pro) and [Baining Guo](https://scholar.google.com/citations?user=h4kYmRYAAAAJ\u0026hl=en\u0026oi=ao).\n\nThis repo is the official implementation of [\"Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation\"](https://arxiv.org/abs/2205.14141).\n\n## Updates\n***11/30/2022***\n\n1. Distilled and fine-tuned models on ImageNet-1K (`ViT Large`) are provided.\n\n***11/28/2022***\n\nInitial commits:\n\n1. Distilled and fine-tuned models on ImageNet-1K (`Swin Base`, and `ViT Base`) are provided.\n2. The supported code for ImageNet-1K distillation and fine-tuning is provided.\n\n## Introduction\n\n**FD** is initially described in [arxiv](https://arxiv.org/abs/2205.14141), which is a simple framework to convert the traditional pre-training models, such as image classification (DeiT), instance contrastive learning (DINO) and image-text alignment (CLIP) into new models with better fine-tuning performances. Through a set of diagosing tools, we find that the models distilled with feature map are endowed with following good properties which are also revealed in masked image modeling models: 1) more diverse attention heads; 2) more diagonal attention patterns; 3) flatten loss landscapes. \n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"figures/teaser.jpg\" height=\"250px\" /\u003e\n\u003c/div\u003e\n\n## Main Results on ImageNet\n\n### Swin Transformer\n\n**ImageNet-1K Distilled and Fine-tuned Models**\n\n| name | distillation epochs | teacher model | image resolution | acc@1 | distilled model | fine-tuned model |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| Swin-Base | 300 | [EsViT-Base](https://github.com/microsoft/esvit) | 224x224 | 85.1 | [google](https://drive.google.com/file/d/11_GQUHgcrUO8PMzl73eJmLSa7f3c5dZY/view?usp=sharing)/[config](configs/pretrain/fd_pretrain__esvit_swin_base__img224__300ep.yaml) | [google](https://drive.google.com/file/d/1criliGcjpEJxqlsYRGBERBAMYrFYFW--/view?usp=sharing)/[config](configs/finetune/fd_finetune__esvit_swin_base__img224__300ep.yaml) |\n\n### Vision Transformer\n\n**ImageNet-1K Distilled and Fine-tuned Models**\n\n| name | distillation epochs | teacher model | image resolution | acc@1 | distilled model | fine-tuned model |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| ViT-Base | 300 | [CLIP-Base](https://github.com/openai/CLIP) | 224x224 | 84.9 | [google](https://drive.google.com/file/d/1XFOZ6rJkv5X08Bu5d04_Xy3iJOj6SLc7/view?usp=sharing)/[config](configs/pretrain/fd_pretrain__clip_vit_base__img224__300ep.yaml) | [google](https://drive.google.com/file/d/1mP_JESmcdFeIkpB4aYyFzALtkydy_9qN/view?usp=sharing)/[config](configs/finetune/fd_finetune__clip_vit_base__img224__300ep.yaml) |\n| ViT-Base | 300 | [DINO-Base](https://github.com/facebookresearch/dino) | 224x224 | 83.8 | [google](https://drive.google.com/file/d/1fwBINMxpv5zFOI7Ye6l9msI8GzocpA3z/view?usp=sharing)/[config](configs/pretrain/fd_pretrain__dino_vit_base__img224__300ep.yaml) | [google](https://drive.google.com/file/d/1Mn_GgepfZXOe7W0UqEQMFo5MjJpMwM_i/view?usp=sharing)/[config](configs/finetune/fd_finetune__dino_vit_base__img224__300ep.yaml) |\n| ViT-Base | 300 | [DeiT-Base](https://github.com/facebookresearch/deit) | 224x224 | 83.0 | [google](https://drive.google.com/file/d/1yPezioDc4O6hdfD6VSAIU9DvJiXG4ZSJ/view?usp=sharing)/[config](configs/pretrain/fd_pretrain__deit_vit_base__img224__300ep.yaml) | [google](https://drive.google.com/file/d/1pb0KUlVcCaEGT-xnx6ookrqcC-88Ori5/view?usp=sharing)/[config](configs/finetune/fd_finetune__deit_vit_base__img224__300ep.yaml) |\n| ViT-Large | 300 | [CLIP-Large](https://github.com/openai/CLIP) | 224x224 | 87.7 | [google](https://drive.google.com/file/d/1H5USyzqwoS31JHDX874q8a70LdVD9zNY/view?usp=sharing)/[config](configs/pretrain/fd_pretrain__clip_vit_large__img224__300ep.yaml) | [google](https://drive.google.com/file/d/1XDDbDl9jzt8H2Fy6iZNfNA7Yjepf_MGx/view?usp=sharing)/[config](configs/finetune/fd_finetune__clip_vit_large__img224__300ep.yaml) |\n\n## Citation\n\nIf you find our work useful in your research, please cite:\n\n```\n@article{wei2022FD,\n  title={Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation},\n  author={Yixuan Wei and Han Hu and Zhenda Xie and Zheng Zhang and Yue Cao and Jianmin Bao and Dong Chen and Baining Guo},\n  journal={Tech Report},\n  year={2022}\n}\n```\n\n## Getting Started\n\n### Installation\n\n- Install `CUDA 11.3` with `cuDNN 8` following the official installation guide of [CUDA](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html) and [cuDNN](https://developer.nvidia.com/rdp/cudnn-archive).\n\n- Setup conda environment:\n```bash\n# Create environment\nconda create -n FD python=3.8 -y\nconda activate FD\n\n# Install requirements\npip install torch==1.12.0+cu113 torchvision==0.13.0+cu113 torchaudio==0.12.0 --extra-index-url https://download.pytorch.org/whl/cu113\n\n# Clone codes\ngit clone https://github.com/SwinTransformer/Feature-Distillation\ncd Feature-Distillation\n\n# Install other requirements\npip install -r requirements.txt\n```\n\n### Feature-Distillation\nTo distill models, run:\n```bash\npython -m torch.distributed.launch --nproc_per_node \u003cnum-of-gpus-to-use\u003e main_fd.py \\ \n--cfg \u003cconfig-file\u003e --data-path \u003cimagenet-path\u003e/train [--batch-size \u003cbatch-size-per-gpu\u003e --output \u003coutput-directory\u003e --tag \u003cjob-tag\u003e]\n```\n\nFor example, to distill `CLIP-Base` for 300 epochs on one DGX-2 server, run:\n```bash\npython -m torch.distributed.launch --nproc_per_node=16 main_fd.py --cfg configs/pretrain/fd_pretrain__clip_vit_base__img224__300ep.yaml --batch-size 128 --data-path \u003cimagenet-path\u003e/train [--output \u003coutput-directory\u003e --tag \u003cjob-tag\u003e]\n```\n\nIf you want to save gpu memory consumption, add `--use-checkpoint`.\n\n### Fine-tuning distilled models\nTo fine-tune distilled models, run:\n```bash\npython -m torch.distributed.launch --nproc_per_node \u003cnum-of-gpus-to-use\u003e main_finetune.py \\ \n--cfg \u003cconfig-file\u003e --data-path \u003cimagenet-path\u003e --pretrained \u003cpretrained-ckpt\u003e [--batch-size \u003cbatch-size-per-gpu\u003e --output \u003coutput-directory\u003e --tag \u003cjob-tag\u003e]\n```\n\nFor example, to fine-tune `Distilled-CLIP-Base` on one DGX-2 server, run:\n```bash\npython -m torch.distributed.launch --nproc_per_node 16 main_finetune.py \\ \n--cfg configs/finetune/fd_finetune__clip_vit_base__img224__300ep.yaml --batch-size 128 --data-path \u003cimagenet-path\u003e --pretrained \u003cpretrained-ckpt\u003e [--output \u003coutput-directory\u003e --tag \u003cjob-tag\u003e]\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSwinTransformer%2FFeature-Distillation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FSwinTransformer%2FFeature-Distillation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSwinTransformer%2FFeature-Distillation/lists"}