{"id":18906994,"url":"https://github.com/jordandeklerk/vit","last_synced_at":"2026-07-22T13:32:53.962Z","repository":{"id":216756220,"uuid":"741785490","full_name":"jordandeklerk/ViT","owner":"jordandeklerk","description":"Implementing a vision transformer model in PyTorch on CIFAR-10","archived":false,"fork":false,"pushed_at":"2024-02-28T20:13:50.000Z","size":6438,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-22T09:59:13.455Z","etag":null,"topics":["artificial-intelligence","attention-mechanism","computer-vision","image-classification","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jordandeklerk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-11T05:26:32.000Z","updated_at":"2025-02-13T15:00:23.000Z","dependencies_parsed_at":"2024-11-08T09:30:41.504Z","dependency_job_id":null,"html_url":"https://github.com/jordandeklerk/ViT","commit_stats":null,"previous_names":["jordandeklerk/vit-pytorch","jordandeklerk/vit"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jordandeklerk/ViT","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jordandeklerk%2FViT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jordandeklerk%2FViT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jordandeklerk%2FViT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jordandeklerk%2FViT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jordandeklerk","download_url":"https://codeload.github.com/jordandeklerk/ViT/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jordandeklerk%2FViT/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35764375,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-07-22T02:00:06.236Z","response_time":124,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","attention-mechanism","computer-vision","image-classification","transformers"],"created_at":"2024-11-08T09:19:27.628Z","updated_at":"2026-07-22T13:32:53.935Z","avatar_url":"https://github.com/jordandeklerk.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Vision Transformer on CIFAR-10\n\n\u003chr\u003e\n\n## Contents\n\n1. [Highlights](#Highlights)\n2. [Requirements](#Requirements)\n3. [Usage](#Usage)\n4. [Results](#Results)\n\n\n\u003chr\u003e\n\n## Highlights\nThis project is a implementation from scratch of a slightly modified version of the vanilla vision transformer introduced in the paper [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://arxiv.org/abs/2010.11929). We implement this model on the small scale benchmark dataset `CIFAR-10`. \n\nVision Transformers often suffer when trained from scratch on small datasets such as `CIFAR-10`. This is primarily due to the lack of locality, inductive biases and hierarchical structure of the representations which is commonly observed in the Convolutional Neural Networks. As a result, ViTs require large-scale pre-training to learn such properties from the data for better transfer learning to downstream tasks. \n\nThis project shows that with modifications, supervised training of vision transformer models on small scale datasets like `CIFAR-10` can lead to very high accuracy with low computational constraints. \n\n\u003cimg src=\"./Images/vit.gif\" width=\"750\"\u003e\u003c/img\u003e\n\nThe vanilla vision transformer model uses the standard multi-head self-attention mechanism introduced in the seminal paper by [Vaswani et al.](https://arxiv.org/abs/1706.03762).\n\n```python\nclass Attention(nn.Module):\n    def __init__(self, dim, num_heads=8, qkv_bias=False, qk_scale=None, attn_drop=0., proj_drop=0.):\n        super().__init__()\n        self.num_heads = num_heads\n        head_dim = dim // num_heads\n        self.scale = qk_scale or head_dim ** -0.5\n        all_head_dim = head_dim * self.num_heads\n        self.qkv = nn.Linear(dim, dim * 3, bias=qkv_bias)\n        self.attn_drop = nn.Dropout(attn_drop)\n        self.proj = nn.Linear(all_head_dim, dim)\n        self.proj_drop = nn.Dropout(proj_drop)\n\n    def forward(self, x):\n        B, N, C = x.shape\n        \n        qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, C // self.num_heads).permute(2, 0, 3, 1, 4)\n        q, k, v = qkv[0], qkv[1], qkv[2]\n\n        attn = (q @ k.transpose(-2, -1)) * self.scale\n        attn = attn.softmax(dim=-1)\n        attn = self.attn_drop(attn)\n\n        x = (attn @ v).transpose(1, 2).reshape(B, N, C)\n        x = self.proj(x)\n        x = self.proj_drop(x)\n        return x, attn\n```\n\n\n\u003chr\u003e\n\n## Requirements\n```shell\npip install -r requirements.txt\n```\n\n\u003chr\u003e\n\n## Usage\nTo replicate the reported results, clone this repo\n```shell\ncd your_directory git clone git@github.com:jordandeklerk/ViT-pytorch.git\n```\nand run the main training script\n```shell\npython train.py \n```\nMake sure to adjust the checkpoint directory in `train.py` to store checkpoint files.\n\n\u003chr\u003e\n\n## Results\nWe test our approach on the `CIFAR-10` dataset with the intention to extend our model to 4 other small low resolution datasets: `Tiny-Imagenet`, `CIFAR100`, `CINIC10` and `SVHN`. All training took place on a single A100 GPU.\n  * CIFAR10\n    * ```vit_cifar10_patch2_input32``` - 96.80 @ 32\n    \n\nFlop analysis:\n```\ntotal flops: 915674304\ntotal activations: 10735212\nnumber of parameter: 2725632\n| module            | #parameters or shape   | #flops   |\n|:------------------|:-----------------------|:---------|\n| model             | 2.726M                 | 0.916G   |\n|  cls_token        |  (1, 1, 192)           |          |\n|  pos_embed        |  (1, 257, 192)         |          |\n|  patch_embed.proj |  2.496K                |  0.59M   |\n|  blocks           |  2.673M                |  0.915G  |\n|  norm             |  0.384K                |  0.247M  |\n```\n\n\u003chr\u003e\n\n## Citations\n```bibtex\n@misc{vaswani2017attention,\n    title   = {Attention Is All You Need},\n    author  = {Ashish Vaswani and Noam Shazeer and Niki Parmar and Jakob Uszkoreit and Llion Jones and Aidan N. Gomez and Lukasz Kaiser and Illia Polosukhin},\n    year    = {2017},\n    eprint  = {1706.03762},\n    archivePrefix = {arXiv},\n    primaryClass = {cs.CL}\n}\n```\n\n```bibtex\n@misc{dosovitskiy2020image,\n    title   = {An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale},\n    author  = {Alexey Dosovitskiy and Lucas Beyer and Alexander Kolesnikov and Dirk Weissenborn and Xiaohua Zhai and Thomas Unterthiner and Mostafa Dehghani and Matthias Minderer and Georg Heigold and Sylvain Gelly and Jakob Uszkoreit and Neil Houlsby},\n    year    = {2020},\n    eprint  = {2010.11929},\n    archivePrefix = {arXiv},\n    primaryClass = {cs.CV}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjordandeklerk%2Fvit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjordandeklerk%2Fvit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjordandeklerk%2Fvit/lists"}