https://github.com/wangkai930418/attndistill
code for our paper "Attention Distillation: self-supervised vision transformer students need more guidance" in BMVC 2022
https://github.com/wangkai930418/attndistill
bmvc knowledge-distillation self-supervised-learning teacher-student-learning vision-transformers
Last synced: about 1 year ago
JSON representation
code for our paper "Attention Distillation: self-supervised vision transformer students need more guidance" in BMVC 2022
- Host: GitHub
- URL: https://github.com/wangkai930418/attndistill
- Owner: wangkai930418
- Created: 2022-10-01T12:41:54.000Z (almost 4 years ago)
- Default Branch: master
- Last Pushed: 2022-10-04T07:44:05.000Z (almost 4 years ago)
- Last Synced: 2025-03-27T06:05:31.994Z (over 1 year ago)
- Topics: bmvc, knowledge-distillation, self-supervised-learning, teacher-student-learning, vision-transformers
- Language: Python
- Homepage:
- Size: 29.3 KB
- Stars: 16
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: readme.md
Awesome Lists containing this project
README
# **[Attention Distillation: self-supervised vision transformer students need more guidance (BMVC 2022)](https://arxiv.org/abs/2210.00944)**
[Kai Wang](https://scholar.google.com/citations?user=j14vd0wAAAAJ), [Fei Yang](https://scholar.google.com/citations?user=S1gksNwAAAAJ&hl=en) and [Joost van de Weijer](https://scholar.google.com/citations?user=Gsw2iUEAAAAJ&hl=en)
## Requirements
Please check the packages in your environment with the "requirements.txt", normally these scripts don't depend on much on the package versions.
## Reproducing
Please modify the data paths in "dino_teacher.sh" and "mugs_teacher.sh" to run the scripts by "bash xxx.sh".
## Download
The teacher models checkpoints can be downloaded from the github repositories of DINO (https://github.com/facebookresearch/dino) and Mugs (https://github.com/sail-sg/mugs).
## Others
If you have any question, do not hesitate to contact me or post an issue.