{"id":44626208,"url":"https://github.com/cmsflash/efficient-attention","last_synced_at":"2026-02-14T15:13:14.064Z","repository":{"id":43599813,"uuid":"167199096","full_name":"cmsflash/efficient-attention","owner":"cmsflash","description":"An implementation of the efficient attention module.","archived":false,"fork":false,"pushed_at":"2020-11-30T06:06:12.000Z","size":1455,"stargazers_count":244,"open_issues_count":1,"forks_count":23,"subscribers_count":6,"default_branch":"master","last_synced_at":"2023-11-07T14:31:57.213Z","etag":null,"topics":["attention-mechanism","computer-vision","deep-learning","paper","paper-implementation","paper-open-source"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/1812.01243","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cmsflash.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-01-23T14:44:40.000Z","updated_at":"2023-10-27T03:41:18.000Z","dependencies_parsed_at":"2022-08-12T10:41:53.933Z","dependency_job_id":null,"html_url":"https://github.com/cmsflash/efficient-attention","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"purl":"pkg:github/cmsflash/efficient-attention","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmsflash%2Fefficient-attention","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmsflash%2Fefficient-attention/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmsflash%2Fefficient-attention/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmsflash%2Fefficient-attention/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cmsflash","download_url":"https://codeload.github.com/cmsflash/efficient-attention/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cmsflash%2Fefficient-attention/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29447774,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-14T14:10:32.461Z","status":"ssl_error","status_checked_at":"2026-02-14T14:09:49.945Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention-mechanism","computer-vision","deep-learning","paper","paper-implementation","paper-open-source"],"created_at":"2026-02-14T15:13:12.526Z","updated_at":"2026-02-14T15:13:14.052Z","avatar_url":"https://github.com/cmsflash.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Efficient Attention\n\nAn implementation of the [efficient attention](https://arxiv.org/abs/1812.01243) module.\n\n## Description\n\n![](illustration.png)\n\nEfficient attention is an attention mechanism that substantially optimizes the memory and computational efficiency while retaining **exactly** the same expressive power as the conventional dot-product attention. The illustration above compares the two types of attention. The efficient attention module is a drop-in replacement for the non-local module ([Wang et al., 2018](https://arxiv.org/abs/1711.07971)), while it:\n\n- uses less resources to achieve the same accuracy;\n- achieves higher accuracy with the same resource constraints (by allowing more insertions); and\n- is applicable in domains and models where the non-local module is not (due to resource constraints).\n\n## Resources\n\nYouTube:\n- Presentation: https://youtu.be/_wnjhTM04NM\n\nbilibili (for users in Mainland China):\n- Presentation: https://www.bilibili.com/video/BV1tK4y1f7Rm\n- Presentation in Chinese: https://www.bilibili.com/video/bv1Gt4y1Y7E3 \n\n## Implementation details\n\nThis repository implements the efficient attention module with softmax normalization, output reprojection, and residual connection.\n\n## Features not in the paper\n\nThis repository implements additionally implements the multi-head mechanism which was not in the paper. To learn more about the mechanism, refer to [Vaswani et al.](https://arxiv.org/abs/1706.03762)\n\n## Citation\n\nThe [paper](https://arxiv.org/abs/1812.01243) will appear at WACV 2021. If you use, compare with, or refer to this work, please cite\n\n```bibtex\n@inproceedings{shen2021efficient,\n    author = {Zhuoran Shen and Mingyuan Zhang and Haiyu Zhao and Shuai Yi and Hongsheng Li},\n    title = {Efficient Attention: Attention with Linear Complexities},\n    booktitle = {WACV},\n    year = {2021},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcmsflash%2Fefficient-attention","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcmsflash%2Fefficient-attention","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcmsflash%2Fefficient-attention/lists"}