{"id":15601027,"url":"https://github.com/lucidrains/deformable-attention","last_synced_at":"2025-05-15T12:06:57.237Z","repository":{"id":41194090,"uuid":"470848943","full_name":"lucidrains/deformable-attention","owner":"lucidrains","description":"Implementation of Deformable Attention in Pytorch from the paper \"Vision Transformer with Deformable Attention\"","archived":false,"fork":false,"pushed_at":"2025-02-03T21:34:57.000Z","size":150,"stargazers_count":334,"open_issues_count":4,"forks_count":33,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-04-14T20:57:48.463Z","etag":null,"topics":["artificial-intelligence","attention-mechanism","deep-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lucidrains.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-03-17T04:35:19.000Z","updated_at":"2025-04-14T07:31:46.000Z","dependencies_parsed_at":"2025-03-31T16:08:09.126Z","dependency_job_id":"72ec6808-49e0-4d79-bedd-e8db93710d30","html_url":"https://github.com/lucidrains/deformable-attention","commit_stats":{"total_commits":35,"total_committers":2,"mean_commits":17.5,"dds":0.02857142857142858,"last_synced_commit":"86dad2adb8833c7bdad1dfe40a614e2a4f56c9d8"},"previous_names":[],"tags_count":20,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fdeformable-attention","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fdeformable-attention/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fdeformable-attention/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fdeformable-attention/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lucidrains","download_url":"https://codeload.github.com/lucidrains/deformable-attention/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254337616,"owners_count":22054254,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","attention-mechanism","deep-learning"],"created_at":"2024-10-03T02:12:31.369Z","updated_at":"2025-05-15T12:06:52.225Z","avatar_url":"https://github.com/lucidrains.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg src=\"./deformable-attention.png\" width=\"500px\"\u003e\u003c/img\u003e\n\n## Deformable Attention\n\nImplementation of Deformable Attention from \u003ca href=\"https://arxiv.org/abs/2201.00520\"\u003ethis paper\u003c/a\u003e in Pytorch, which appears to be an improvement to what was proposed in DETR. The relative positional embedding has also been modified for better extrapolation, using the Continuous Positional Embedding proposed in SwinV2.\n\n## Install\n\n```bash\n$ pip install deformable-attention\n```\n\n## Usage\n\n```python\nimport torch\nfrom deformable_attention import DeformableAttention\n\nattn = DeformableAttention(\n    dim = 512,                   # feature dimensions\n    dim_head = 64,               # dimension per head\n    heads = 8,                   # attention heads\n    dropout = 0.,                # dropout\n    downsample_factor = 4,       # downsample factor (r in paper)\n    offset_scale = 4,            # scale of offset, maximum offset\n    offset_groups = None,        # number of offset groups, should be multiple of heads\n    offset_kernel_size = 6,      # offset kernel size\n)\n\nx = torch.randn(1, 512, 64, 64)\nattn(x) # (1, 512, 64, 64)\n```\n\n3d deformable attention\n\n```python\nimport torch\nfrom deformable_attention import DeformableAttention3D\n\nattn = DeformableAttention3D(\n    dim = 512,                          # feature dimensions\n    dim_head = 64,                      # dimension per head\n    heads = 8,                          # attention heads\n    dropout = 0.,                       # dropout\n    downsample_factor = (2, 8, 8),      # downsample factor (r in paper)\n    offset_scale = (2, 8, 8),           # scale of offset, maximum offset\n    offset_kernel_size = (4, 10, 10),   # offset kernel size\n)\n\nx = torch.randn(1, 512, 10, 32, 32) # (batch, dimension, frames, height, width)\nattn(x) # (1, 512, 10, 32, 32)\n```\n\n1d deformable attention for good measure\n\n```python\nimport torch\nfrom deformable_attention import DeformableAttention1D\n\nattn = DeformableAttention1D(\n    dim = 128,\n    downsample_factor = 4,\n    offset_scale = 2,\n    offset_kernel_size = 6\n)\n\nx = torch.randn(1, 128, 512)\nattn(x) # (1, 128, 512)\n```\n\n## Citation\n\n```bibtex\n@misc{xia2022vision,\n    title   = {Vision Transformer with Deformable Attention}, \n    author  = {Zhuofan Xia and Xuran Pan and Shiji Song and Li Erran Li and Gao Huang},\n    year    = {2022},\n    eprint  = {2201.00520},\n    archivePrefix = {arXiv},\n    primaryClass = {cs.CV}\n}\n```\n\n```bibtex\n@misc{liu2021swin,\n    title   = {Swin Transformer V2: Scaling Up Capacity and Resolution},\n    author  = {Ze Liu and Han Hu and Yutong Lin and Zhuliang Yao and Zhenda Xie and Yixuan Wei and Jia Ning and Yue Cao and Zheng Zhang and Li Dong and Furu Wei and Baining Guo},\n    year    = {2021},\n    eprint  = {2111.09883},\n    archivePrefix = {arXiv},\n    primaryClass = {cs.CV}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Fdeformable-attention","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flucidrains%2Fdeformable-attention","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Fdeformable-attention/lists"}