{"id":15650157,"url":"https://github.com/sayakpaul/deit-tf","last_synced_at":"2025-04-30T16:43:59.331Z","repository":{"id":42467968,"uuid":"474556959","full_name":"sayakpaul/deit-tf","owner":"sayakpaul","description":"Includes PyTorch -\u003e Keras model porting code for DeiT models with fine-tuning and inference notebooks.","archived":false,"fork":false,"pushed_at":"2022-04-30T04:52:10.000Z","size":42395,"stargazers_count":42,"open_issues_count":0,"forks_count":7,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-09T23:22:40.861Z","etag":null,"topics":["computer-vision","distillation","image-recognition","imagenet-1k","inductive-biases","keras","tensorflow","vision-transformers"],"latest_commit_sha":null,"homepage":"https://tfhub.dev/sayakpaul/collections/deit/1","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sayakpaul.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-03-27T06:42:17.000Z","updated_at":"2025-01-08T23:51:48.000Z","dependencies_parsed_at":"2022-09-08T20:30:39.334Z","dependency_job_id":null,"html_url":"https://github.com/sayakpaul/deit-tf","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sayakpaul%2Fdeit-tf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sayakpaul%2Fdeit-tf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sayakpaul%2Fdeit-tf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sayakpaul%2Fdeit-tf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sayakpaul","download_url":"https://codeload.github.com/sayakpaul/deit-tf/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233686583,"owners_count":18714175,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","distillation","image-recognition","imagenet-1k","inductive-biases","keras","tensorflow","vision-transformers"],"created_at":"2024-10-03T12:33:37.202Z","updated_at":"2025-01-13T03:20:01.465Z","avatar_url":"https://github.com/sayakpaul.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DeiT-TF (Data-efficient Image Transformers)\n\nThis repository provides TensorFlow / Keras implementations of different DeiT\n[1] variants from Touvron et al. It also provides the TensorFlow / Keras models that have been\npopulated with the original DeiT pre-trained params available from [2]. These\nmodels are not blackbox SavedModels i.e., they can be fully expanded into `tf.keras.Model`\nobjects and one can call all the utility functions on them (example: `.summary()`).\n\nAs of today, all the TensorFlow / Keras variants of the **DeiT** models listed\n[here](https://github.com/facebookresearch/deit#model-zoo) are available in this\nrepository.\n\nRefer to the [\"Using the models\"](https://github.com/sayakpaul/deit-tf#using-the-models)\nsection to get started. You can also follow along with this tutorial: https://keras.io/examples/vision/deit/.\n\n**Updates**\n\n* April 29, 2022: This project received the [Google OSS Expert prize](https://www.kaggle.com/discussions/general/322041).\n* April 22, 2022: This project won the [#TFCommunitySpotlight award](https://twitter.com/TensorFlow/status/1516869315517198337).\n\n## Table of contents\n\n* [Conversion](https://github.com/sayakpaul/deit-tf#conversion)\n* [Collection of pre-trained models (converted from PyTorch to TensorFlow)](https://github.com/sayakpaul/deit-tf#models)\n* [Results of the converted models](https://github.com/sayakpaul/deit-tf#results)\n* [How to use the models?](https://github.com/sayakpaul/deit-tf#using-the-models)\n* [Training with DeiT](https://github.com/sayakpaul/deit-tf#training-with-deit)\n* [References](https://github.com/sayakpaul/deit-tf#references)\n* [Acknowledgements](https://github.com/sayakpaul/deit-tf#acknowledgements)\n\n## Conversion\n\nTensorFlow / Keras implementations are available in `vit/vit_models.py` and `vit/deit_models.py`.\nConversion utilities are in `convert.py`.\n\n## Models\n\nFind the models on TF-Hub here: https://tfhub.dev/sayakpaul/collections/deit/1. You can fully inspect the\narchitecture of the TF-Hub models like so:\n\n```py\nimport tensorflow as tf\n\nmodel_gcs_path = \"gs://tfhub-modules/sayakpaul/deit_tiny_patch16_224/1/uncompressed\"\nmodel = tf.keras.models.load_model(model_gcs_path)\n\ndummy_inputs = tf.ones((2, 224, 224, 3))\n_ = model(dummy_inputs)\nprint(model.summary(expand_nested=True))\n```\n\n## Results\n\nResults are on ImageNet-1k validation set (top-1 accuracy). \n\n|    | **model_name**                   |   **top1_acc(%)** |   **top5_acc(%)** |   **orig_top1_acc(%)** |   **orig_top5_acc(%)** |\n|---:|:---------------------------------|--------------:|--------------:|-------------------:|-------------------:|\n|  0 | deit_tiny_patch16_224            |        72.136 |        91.128 |               72.2 |               91.1 |\n|  1 | deit_tiny_distilled_patch16_224  |        74.522 |        91.896 |               74.5 |               91.9 |\n|  2 | deit_small_patch16_224           |        79.828 |        94.954 |               79.9 |               95   |\n|  3 | deit_small_distilled_patch16_224 |        81.172 |        95.414 |               81.2 |               95.4 |\n|  4 | deit_base_patch16_224            |        81.798 |        95.592 |               81.8 |               95.6 |\n|  5 | deit_base_patch16_384            |        82.894 |        96.234 |               82.9 |               96.2 |\n|  6 | deit_base_distilled_patch16_224  |        83.326 |        96.496 |               83.4 |               96.5 |\n|  7 | deit_base_distilled_patch16_384  |        85.238 |        97.172 |               85.2 |               97.2 |\n\nResults can be verified with the code in `i1k_eval`. Original results were sourced from [2].\n\n\n## Using the models\n\n**Pre-trained models**:\n\n* Off-the-shelf classification: [Colab Notebook](https://colab.research.google.com/github/sayakpaul/deit-tf/blob/main/notebooks/classification.ipynb)\n* Fine-tuning: [Colab Notebook](https://colab.research.google.com/github/sayakpaul/deit-tf/blob/main/notebooks/finetune.ipynb)\n\nThese models also output attention weights from each of the Transformer blocks.\nRefer to [this notebook](https://colab.research.google.com/github/sayakpaul/deit-tf/blob/main/notebooks/classification.ipynb)\nfor more details. Additionally, the notebook shows how to visualize the attention maps for a given image.\n\n\u003cbr\u003e\n\n\u003cdiv align=center\u003e\n \u003cimg src=\"https://i.ibb.co/hHzggDr/attention-map.png\" width=700/\u003e\n\u003c/div\u003e\n\n\u003cbr\u003e\n \n**Randomly initialized models**:\n \n```py\nfrom vit.model_configs import base_config\nfrom vit.deit_models import ViTDistilled\n\nimport tensorflow as tf\n \ndistilled_tiny_tf_config = base_config.get_config(\n    name=\"deit_tiny_distilled_patch16_224\"\n)\ndeit_tiny_distilled_patch16_224 = ViTDistilled(distilled_tiny_tf_config)\n\ndummy_inputs = tf.ones((2, 224, 224, 3))\n_ = deit_tiny_distilled_patch16_224(dummy_inputs)\nprint(deit_tiny_distilled_patch16_224.summary(expand_nested=True))\n```\n\nTo initialize a network with say, 5 classes do:\n\n```py\nwith distilled_tiny_tf_config.unlocked():\n    distilled_tiny_tf_config.num_classes = 5\ndeit_tiny_distilled_patch16_224 = ViTDistilled(distilled_tiny_tf_config)\n```\n\nTo view different model configurations, refer to `convert_all_models.py`.\n\n\n## Training with DeiT\n\nYou can refer to the `notebooks/deit-trainer.ipynb` notebok to get a sense of how distillation\nis actually performed using DeiT. Additionally, that notebook also provides code in case you\nwanted to train a model from scratch instead of distillation.\n \n\n## References\n\n[1] DeiT paper: https://arxiv.org/abs/2012.12877\n\n[2] Official DeiT code: https://github.com/facebookresearch/deit\n\n## Acknowledgements\n\n* [Aritra Roy Gosthipaty](https://github.com/ariG23498) who was instrumental in implementing\nsome parts of the core ViT module (`vit.vit_models`) for another project.\n* [`timm` library source code](https://github.com/rwightman/pytorch-image-models)\nfor the awesome codebase.\n* [ML-GDE program](https://developers.google.com/programs/experts/) for\nproviding GCP credits that supported my experiments.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsayakpaul%2Fdeit-tf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsayakpaul%2Fdeit-tf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsayakpaul%2Fdeit-tf/lists"}