{"id":15685394,"url":"https://github.com/sayakpaul/cait-tf","last_synced_at":"2026-03-03T04:13:26.458Z","repository":{"id":40541740,"uuid":"484972397","full_name":"sayakpaul/cait-tf","owner":"sayakpaul","description":"Implementation of CaiT models in TensorFlow and ImageNet-1k checkpoints. Includes code for inference and fine-tuning.","archived":false,"fork":false,"pushed_at":"2023-06-09T10:39:43.000Z","size":2611,"stargazers_count":12,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-23T22:03:33.554Z","etag":null,"topics":["class-attention","deeper-vits","image-recognition","imagenet-1k","self-attention","tensorflow","vision-transformers"],"latest_commit_sha":null,"homepage":"https://tfhub.dev/sayakpaul/collections/cait/1","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sayakpaul.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-24T08:42:39.000Z","updated_at":"2024-01-08T06:48:02.000Z","dependencies_parsed_at":"2024-10-23T18:11:36.345Z","dependency_job_id":null,"html_url":"https://github.com/sayakpaul/cait-tf","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sayakpaul/cait-tf","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sayakpaul%2Fcait-tf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sayakpaul%2Fcait-tf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sayakpaul%2Fcait-tf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sayakpaul%2Fcait-tf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sayakpaul","download_url":"https://codeload.github.com/sayakpaul/cait-tf/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sayakpaul%2Fcait-tf/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30031981,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-03T03:27:35.548Z","status":"ssl_error","status_checked_at":"2026-03-03T03:27:09.213Z","response_time":61,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["class-attention","deeper-vits","image-recognition","imagenet-1k","self-attention","tensorflow","vision-transformers"],"created_at":"2024-10-03T17:25:04.936Z","updated_at":"2026-03-03T04:13:26.444Z","avatar_url":"https://github.com/sayakpaul.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CaiT-TF (Going deeper with Image Transformers)\n\n[![TensorFlow 2.8](https://img.shields.io/badge/TensorFlow-2.8-FF6F00?logo=tensorflow)](https://github.com/tensorflow/tensorflow/releases/tag/v2.8.0)\n[![HugginFace badge](https://img.shields.io/badge/🤗%20Hugging%20Face-Spaces-yellow.svg)](https://huggingface.co/spaces)\n[![Models on TF-Hub](https://img.shields.io/badge/TF--Hub-Models%20on%20TF--Hub-orange)](https://tfhub.dev/sayakpaul/collections/cait/1)\n\nThis repository provides TensorFlow / Keras implementations of different CaiT\n[1] variants from Touvron et al. It also provides the TensorFlow / Keras models that have been\npopulated with the original CaiT pre-trained params available from [2]. These\nmodels are not blackbox SavedModels i.e., they can be fully expanded into `tf.keras.Model`\nobjects and one can call all the utility functions on them (example: `.summary()`).\n\nAs of today, all the TensorFlow / Keras variants of the **CaiT** models listed\n[here](https://github.com/facebookresearch/deit/blob/main/README_cait.md#model-zoo) are\navailable in this repository. \n\nRefer to the [\"Using the models\"](https://github.com/sayakpaul/cait-tf#using-the-models)\nsection to get started. \n\n**Updates Oct 8 2022**: This project [received](https://www.kaggle.com/discussions/general/358450)\nthe Kaggle ML Research Spotlight Prize (September 2022)\n\n**Updates Sept 25 2022**: [Blog post on CaiT](https://keras.io/examples/vision/cait/)\n\n## Table of contents\n\n* [Conversion](https://github.com/sayakpaul/cait-tf#conversion)\n* [Collection of pre-trained models (converted from PyTorch to TensorFlow)](https://github.com/sayakpaul/cait-tf#models)\n* [Results of the converted models](https://github.com/sayakpaul/cait-tf#results)\n* [How to use the models?](https://github.com/sayakpaul/cait-tf#using-the-models)\n* [References](https://github.com/sayakpaul/cait-tf#references)\n* [Acknowledgements](https://github.com/sayakpaul/cait-tf#acknowledgements)\n\n## Conversion\n\nTensorFlow / Keras implementations are available in `cait/models.py`. Conversion\nutilities are in `convert.py`.\n\n## Models\n\nFind the models on TF-Hub here: https://tfhub.dev/sayakpaul/collections/cait/1. You can fully inspect the\narchitecture of the TF-Hub models like so:\n\n```py\nimport tensorflow as tf\n\nmodel_gcs_path = \"gs://tfhub-modules/sayakpaul/cait_xxs24_224/1/uncompressed\"\nmodel = tf.keras.models.load_model(model_gcs_path)\n\ndummy_inputs = tf.ones((2, 224, 224, 3))\n_ = model(dummy_inputs)\nprint(model.summary(expand_nested=True))\n```\n\n## Results\n\nResults are on ImageNet-1k validation set (top-1 and top-5 accuracies). \n\n| model_name     |   top1_acc(%) |   top5_acc(%) |\n|:---------------:|:--------------:|:--------------:|\n| cait_s24_224   |        83.368 |        96.576 |\n| cait_xxs24_224 |        78.524 |        94.212 |\n| cait_xxs36_224 |        79.76  |        94.876 |\n| cait_xxs36_384 |        81.976 |        96.064 |\n| cait_xxs24_384 |        80.648 |        95.516 |\n| cait_xs24_384  |        83.738 |        96.756 |\n| cait_s24_384   |        84.944 |        97.212 |\n| cait_s36_384   |        85.192 |        97.372 |\n| cait_m36_384   |        85.924 |        97.598 |\n| cait_m48_448   |        86.066 |        97.590 |\n\n\nResults can be verified with the code in `i1k_eval`. Results are in line with [1].\n[Slight differences](https://github.com/facebookresearch/deit/blob/main/README_cait.md#model-zoo) in the results stemmed\nfrom the fact that I used a different set of augmentation transformations. Original \ntransformations suggested by the authors can be found [here](https://github.com/facebookresearch/deit/blob/main/README_cait.md).\n\n\n## Using the models\n\n**Pre-trained models**:\n\n* Off-the-shelf classification: [Colab Notebook](https://colab.research.google.com/github/sayakpaul/cait-tf/blob/main/notebooks/classification.ipynb)\n* Fine-tuning: [Colab Notebook](https://colab.research.google.com/github/sayakpaul/cait-tf/blob/main/notebooks/finetune.ipynb)\n\nThese models also output attention weights from each of the Transformer blocks.\nRefer to [this notebook](https://colab.research.google.com/github/sayakpaul/cait-tf/blob/main/notebooks/classification.ipynb)\nfor more details. Additionally, the notebook shows how to visualize the attention maps for a given image (following\nfigures 6 and 7 of the original paper).\n\n| Original Image | Class Attention Maps | Class Saliency Map |\n| :--: | :--: | :--: |\n| ![cropped image](./assets/butterfly_cropped.png) | ![cls attn](./assets/cls_attn_heads.png) | ![saliency](./assets/cls_saliency.png) |\n\nFor the best quality, refer to the `assets` directory. You can also generate these plots using the following interactive demos on\nHugging Face Spaces:\n\n* [Generate Class Attention plots](https://huggingface.co/spaces/probing-vits/class-attention-map)\n* [Generate Class Saliency plots](https://huggingface.co/spaces/probing-vits/class-saliency)\n \n**Randomly initialized models**:\n \n```py\nfrom cait.model_configs import base_config\nfrom cait.models import CaiT\nimport tensorflow as tf\n \nconfig = base_config.get_config(\n    model_name=\"cait_xxs24_224\"\n)\ncait_xxs24_224 = CaiT(config)\n\ndummy_inputs = tf.ones((2, 224, 224, 3))\n_ = cait_xxs24_224(dummy_inputs)\nprint(cait_xxs24_224.summary(expand_nested=True))\n```\n\nTo initialize a network with say, 5 classes, do:\n\n```py\nconfig = base_config.get_config(\n    model_name=\"cait_xxs24_224\"\n)\nwith config.unlocked():\n    config.num_classes = 5\ncait_xxs24_224 = CaiT(config)\n```\n\nTo view different model configurations, refer to `convert_all_models.py`.\n\n## References\n\n[1] CaiT paper: https://arxiv.org/abs/2103.17239\n\n[2] Official CaiT code: https://github.com/facebookresearch/deit\n\n## Acknowledgements\n\n* [`timm` library source code](https://github.com/rwightman/pytorch-image-models)\nfor the awesome codebase.\n* [ML-GDE program](https://developers.google.com/programs/experts/) for\nproviding GCP credits that supported my experiments.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsayakpaul%2Fcait-tf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsayakpaul%2Fcait-tf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsayakpaul%2Fcait-tf/lists"}