{"id":18906990,"url":"https://github.com/jordandeklerk/swinvit","last_synced_at":"2026-03-06T06:31:09.731Z","repository":{"id":217375428,"uuid":"742720531","full_name":"jordandeklerk/SwinViT","owner":"jordandeklerk","description":"Modified Swin Transformer model in PyTorch on CIFAR-10 for image classification","archived":false,"fork":false,"pushed_at":"2025-05-01T01:51:35.000Z","size":1877,"stargazers_count":6,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-01T02:41:12.612Z","etag":null,"topics":["artificial-intelligence","attention-mechanism","computer-vision","deep-learning","image-classification","swin-transformer","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jordandeklerk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-01-13T06:50:19.000Z","updated_at":"2025-05-01T01:51:38.000Z","dependencies_parsed_at":"2024-06-25T01:24:36.997Z","dependency_job_id":"8f3797d1-489f-458d-910e-82093e1bf794","html_url":"https://github.com/jordandeklerk/SwinViT","commit_stats":null,"previous_names":["jordandeklerk/swinvit-pytorch","jordandeklerk/swinvit"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jordandeklerk/SwinViT","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jordandeklerk%2FSwinViT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jordandeklerk%2FSwinViT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jordandeklerk%2FSwinViT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jordandeklerk%2FSwinViT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jordandeklerk","download_url":"https://codeload.github.com/jordandeklerk/SwinViT/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jordandeklerk%2FSwinViT/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30164590,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-06T04:43:31.446Z","status":"ssl_error","status_checked_at":"2026-03-06T04:40:30.133Z","response_time":250,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","attention-mechanism","computer-vision","deep-learning","image-classification","swin-transformer","transformers"],"created_at":"2024-11-08T09:19:25.786Z","updated_at":"2026-03-06T06:31:09.479Z","avatar_url":"https://github.com/jordandeklerk.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Swin Transformer on CIFAR-10\n\n## Highlights\n\n\u003cimg src=\"./images/swin3.png\"\u003e\u003c/img\u003e\n\nThis project is an implementation of a slightly modified version of the Swin transformer introduced in the paper [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://arxiv.org/abs/2103.14030). We implement this model on the small scale benchmark dataset `CIFAR-10`. \n\n**Swin Transformer** (the name `Swin` stands for **S**hifted **win**dow) capably serves as a general-purpose backbone for computer vision. It is basically a hierarchical Transformer whose representation is computed with shifted windows. The shifted windowing scheme brings greater efficiency by limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection.\n\nThis project focuses on implementing Swin on an image classification task and shows that with modifications, supervised training of the Swin transformer model on small scale datasets like `CIFAR-10` can lead to very high accuracy with low computational constraints.\n\n## Project Structure\n\n```\n├── main.py\n├── model\n│   └── swin_vit.py\n├── requirements.txt\n└── utils\n    ├── autoaug.py\n    ├── cutmix.py\n    ├── dataloader.py\n    ├── loss.py\n    ├── optimizer.py\n    ├── parser.py\n    ├── random_erasing.py\n    ├── sampler.py\n    ├── scheduler.py\n    ├── train_functions.py\n    ├── transforms.py\n    └── utils.py\n```\n\n## Usage\n\n### Install Dependencies\n\nCreate a virtual environment and clone this repository:\n\n```bash\n# Clone the repository\ngit clone https://github.com/jordandeklerk/SwinViT.git\ncd SwinViT\n\n# Create a virtual environment\npython3 -m venv myenv\n\n# Activate the virtual environment\nsource myenv/bin/activate\n\n# Install the required Python packages\npip install -r requirements.txt\n```\n\n\u003chr\u003e\n\n## Usage\nTo replicate the reported results, run `main.py` with the following hyperparameters:\n\n```bash\npython main.py  --patch_size 2 \\\n                --weight_decay 0.1 \\\n                --batch_size 128 \\\n                --epochs 200 \\\n                --lr 0.001 \\\n                --warmup_epochs 10 \\\n                --min_lr 1e-6 \\\n                --clip_grad 3.0 \n```\n\n## Results\nWe test our approach on the `CIFAR-10` dataset with the intention to extend our model to 4 other small low resolution datasets: `Tiny-Imagenet`, `CIFAR100`, `CINIC10` and `SVHN`. All training took place on a single A100 GPU.\n  * CIFAR10\n    * ```swin_cifar10_patch2_input32``` - 91.10 @ 32\n\nFlop analysis:\n```\ntotal flops: 242759424\ntotal activations: 1296394\nnumber of parameter: 7048612\n| module       | #parameters or shape   | #flops   |\n|:-------------|:-----------------------|:---------|\n| model        | 7.049M                 | 0.243G   |\n|  patch_embed |  1.44K                 |  0.418M  |\n|  layers      |  7.043M                |  0.242G  |\n|  norm        |  0.768K                |  30.72K  |\n|  head        |  3.85K                 |  3.84K   |\n```\n\n## Citation\n```bibtex\n@article{liu2021Swin,\n  title={Swin Transformer: Hierarchical Vision Transformer using Shifted Windows},\n  author={Liu, Ze and Lin, Yutong and Cao, Yue and Hu, Han and Wei, Yixuan and Zhang, Zheng and Lin, Stephen and Guo, Baining},\n  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},\n  year={2021}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjordandeklerk%2Fswinvit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjordandeklerk%2Fswinvit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjordandeklerk%2Fswinvit/lists"}