{"id":13738553,"url":"https://github.com/The-AI-Summer/self-attention-cv","last_synced_at":"2025-05-08T16:34:37.251Z","repository":{"id":37645985,"uuid":"334760721","full_name":"The-AI-Summer/self-attention-cv","owner":"The-AI-Summer","description":"Implementation of various self-attention mechanisms focused on computer vision. Ongoing repository. ","archived":false,"fork":false,"pushed_at":"2021-09-14T13:23:07.000Z","size":298,"stargazers_count":1199,"open_issues_count":1,"forks_count":154,"subscribers_count":19,"default_branch":"main","last_synced_at":"2025-04-12T16:52:42.122Z","etag":null,"topics":["artificial-intelligence","attention","attention-mechanism","deep-learning","machine-learning","machine-learning-algorithms","self-attention","transformer","transformers"],"latest_commit_sha":null,"homepage":"https://theaisummer.com/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/The-AI-Summer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"github":null,"patreon":"aisummer","open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2021-01-31T21:28:59.000Z","updated_at":"2025-04-09T13:37:14.000Z","dependencies_parsed_at":"2022-07-18T03:30:41.076Z","dependency_job_id":null,"html_url":"https://github.com/The-AI-Summer/self-attention-cv","commit_stats":null,"previous_names":["the-ai-summer/self_attention_cv"],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/The-AI-Summer%2Fself-attention-cv","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/The-AI-Summer%2Fself-attention-cv/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/The-AI-Summer%2Fself-attention-cv/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/The-AI-Summer%2Fself-attention-cv/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/The-AI-Summer","download_url":"https://codeload.github.com/The-AI-Summer/self-attention-cv/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253105666,"owners_count":21855073,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","attention","attention-mechanism","deep-learning","machine-learning","machine-learning-algorithms","self-attention","transformer","transformers"],"created_at":"2024-08-03T03:02:26.592Z","updated_at":"2025-05-08T16:34:36.937Z","avatar_url":"https://github.com/The-AI-Summer.png","language":"Python","funding_links":["https://patreon.com/aisummer"],"categories":["Python"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"feat_img.png\"/\u003e\n\u003c/div\u003e\n\n# Self-attention building blocks for computer vision applications in PyTorch\n\nImplementation of self attention mechanisms for computer vision in PyTorch with einsum and einops.\nFocused on computer vision self-attention modules. \n\n#### Install it via pip \n\n```$ pip install self-attention-cv``` \n\nIt would be nice to pre-install pytorch in your environment, in case you don't have a GPU. To run the tests from the terminal \n```$ pytest``` you may need to run ``` export PYTHONPATH=$PATHONPATH:`pwd` ``` before.\n\n\n## Related articles\n- [How Attention works in Deep Learning](https://theaisummer.com/attention/)\n- [How Transformers work in deep learning and NLP](https://theaisummer.com/transformer/)\n- [How the Vision Transformer (ViT) works in 10 minutes: an image is worth 16x16 words](https://theaisummer.com/vision-transformer/)\n- [Understanding einsum for Deep learning: implement a transformer with multi-head self-attention from scratch](https://theaisummer.com/einsum-attention/)\n- [How Positional Embeddings work in Self-Attention](https://theaisummer.com/positional-embeddings/)\n- [Why multi-head self attention works: math, intuitions and 10+1 hidden insights](https://theaisummer.com/self-attention/)\n\n\n## Code Examples\n\n\n#### Multi-head attention\n\n```python\nimport torch\nfrom self_attention_cv import MultiHeadSelfAttention\n\nmodel = MultiHeadSelfAttention(dim=64)\nx = torch.rand(16, 10, 64)  # [batch, tokens, dim]\nmask = torch.zeros(10, 10)  # tokens X tokens\nmask[5:8, 5:8] = 1\ny = model(x, mask)\n```\n\n#### Axial attention\n\n```python\nimport torch\nfrom self_attention_cv import AxialAttentionBlock\nmodel = AxialAttentionBlock(in_channels=256, dim=64, heads=8)\nx = torch.rand(1, 256, 64, 64)  # [batch, tokens, dim, dim]\ny = model(x)\n```\n\n#### Vanilla Transformer Encoder\n```python\nimport torch\nfrom self_attention_cv import TransformerEncoder\nmodel = TransformerEncoder(dim=64,blocks=6,heads=8)\nx = torch.rand(16, 10, 64)  # [batch, tokens, dim]\nmask = torch.zeros(10, 10)  # tokens X tokens\nmask[5:8, 5:8] = 1\ny = model(x,mask)\n```\n#### Vision Transformer with/without ResNet50 backbone for image classification\n```python\nimport torch\nfrom self_attention_cv import ViT, ResNet50ViT\n\nmodel1 = ResNet50ViT(img_dim=128, pretrained_resnet=False, \n                        blocks=6, num_classes=10, \n                        dim_linear_block=256, dim=256)\n# or\nmodel2 = ViT(img_dim=256, in_channels=3, patch_dim=16, num_classes=10,dim=512)\nx = torch.rand(2, 3, 256, 256)\ny = model2(x) # [2,10]\n```\n\n#### A re-implementation of Unet with the Vision Transformer encoder\n\n```python\nimport torch\nfrom self_attention_cv.transunet import TransUnet\na = torch.rand(2, 3, 128, 128)\nmodel = TransUnet(in_channels=3, img_dim=128, vit_blocks=8,\nvit_dim_linear_mhsa_block=512, classes=5)\ny = model(a) # [2, 5, 128, 128]\n```\n\n#### Bottleneck Attention block \n```python\nimport torch\nfrom self_attention_cv.bottleneck_transformer import BottleneckBlock\ninp = torch.rand(1, 512, 32, 32)\nbottleneck_block = BottleneckBlock(in_channels=512, fmap_size=(32, 32), heads=4, out_channels=1024, pooling=True)\ny = bottleneck_block(inp)\n```\n\n\n### Position embeddings are also available\n\n#### 1D Positional Embeddings \n\n```python\nimport torch\nfrom self_attention_cv.pos_embeddings import AbsPosEmb1D,RelPosEmb1D\n\nmodel = AbsPosEmb1D(tokens=20, dim_head=64)\n# batch heads tokens dim_head\nq = torch.rand(2, 3, 20, 64)\ny1 = model(q)\n\nmodel = RelPosEmb1D(tokens=20, dim_head=64, heads=3)\nq = torch.rand(2, 3, 20, 64)\ny2 = model(q)\n```\n\n#### 2D Positional Embeddings\n```python\nimport torch\nfrom self_attention_cv.pos_embeddings import RelPosEmb2D\ndim = 32  # spatial dim of the feat map\nmodel = RelPosEmb2D(\n    feat_map_size=(dim, dim),\n    dim_head=128)\n\nq = torch.rand(2, 4, dim*dim, 128)\ny = model(q)\n```\n\n## Acknowledgments\nThanks to Alex Rogozhnikov [@arogozhnikov](https://github.com/arogozhnikov) for the awesome einops package. \nFor my re-implementations I have studied and borrowed code from many repositories of Phil Wang [@lucidrains](https://github.com/lucidrains). \nBy studying  his code I have managed to grasp self-attention, discover nlp stuff that are never\nreferred in the papers, and learn from his clean coding style.\n\n### Cited as\n\n```\n@article{adaloglou2021transformer,\n    title   = \"Transformers in Computer Vision\",\n    author  = \"Adaloglou, Nikolas\",\n    journal = \"https://theaisummer.com/\",\n    year    = \"2021\",\n    howpublished = {https://github.com/The-AI-Summer/self-attention-cv},\n  }\n```\n \n## References\n\n1. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... \u0026 Polosukhin, I. (2017). Attention is all you need. arXiv preprint arXiv:1706.03762.\n2. Wang, H., Zhu, Y., Green, B., Adam, H., Yuille, A., \u0026 Chen, L. C. (2020, August). Axial-deeplab: Stand-alone axial-attention for panoptic segmentation. In European Conference on Computer Vision (pp. 108-126). Springer, Cham.\n3. Srinivas, A., Lin, T. Y., Parmar, N., Shlens, J., Abbeel, P., \u0026 Vaswani, A. (2021). Bottleneck Transformers for Visual Recognition. arXiv preprint arXiv:2101.11605.  \n4. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., ... \u0026 Houlsby, N. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.\n5. Ramachandran, P., Parmar, N., Vaswani, A., Bello, I., Levskaya, A., \u0026 Shlens, J. (2019). Stand-alone self-attention in vision models. arXiv preprint arXiv:1906.05909.\n6. Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., ... \u0026 Zhou, Y. (2021). Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306.\n7. Wang, S., Li, B., Khabsa, M., Fang, H., \u0026 Ma, H. (2020). Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768.\n8. Bertasius, G., Wang, H., \u0026 Torresani, L. (2021). Is Space-Time Attention All You Need for Video Understanding?. arXiv preprint arXiv:2102.05095.\n9. Shaw, P., Uszkoreit, J., \u0026 Vaswani, A. (2018). Self-attention with relative position representations. arXiv preprint arXiv:1803.02155.\n\n## Support\nIf you really like this repository and find it useful, please consider (★) starring it, so that it can reach a broader audience of like-minded people. It would be highly appreciated :) !\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FThe-AI-Summer%2Fself-attention-cv","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FThe-AI-Summer%2Fself-attention-cv","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FThe-AI-Summer%2Fself-attention-cv/lists"}