{"id":13614870,"url":"https://github.com/lucidrains/linear-attention-transformer","last_synced_at":"2025-05-15T18:09:10.098Z","repository":{"id":37412912,"uuid":"269470254","full_name":"lucidrains/linear-attention-transformer","owner":"lucidrains","description":"Transformer based on a variant of attention that is linear complexity in respect to sequence length","archived":false,"fork":false,"pushed_at":"2024-05-05T23:40:52.000Z","size":36160,"stargazers_count":764,"open_issues_count":10,"forks_count":71,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-05-10T11:16:51.195Z","etag":null,"topics":["artificial-intelligence","attention-mechanism","deep-learning","pytorch","transformer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lucidrains.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-06-04T21:34:56.000Z","updated_at":"2025-05-08T20:48:02.000Z","dependencies_parsed_at":"2024-06-18T21:17:31.219Z","dependency_job_id":"cd047601-e77b-454e-8f44-b4c8d59aa896","html_url":"https://github.com/lucidrains/linear-attention-transformer","commit_stats":{"total_commits":81,"total_committers":1,"mean_commits":81.0,"dds":0.0,"last_synced_commit":"24ecf20b11a7c8ddbc15e33a30f0be0cc73b145d"},"previous_names":["lucidrains/linear-attention"],"tags_count":29,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Flinear-attention-transformer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Flinear-attention-transformer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Flinear-attention-transformer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Flinear-attention-transformer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lucidrains","download_url":"https://codeload.github.com/lucidrains/linear-attention-transformer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254394722,"owners_count":22063984,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","attention-mechanism","deep-learning","pytorch","transformer"],"created_at":"2024-08-01T20:01:06.574Z","updated_at":"2025-05-15T18:09:05.089Z","avatar_url":"https://github.com/lucidrains.png","language":"Python","funding_links":[],"categories":["Transformer"],"sub_categories":["Repositories"],"readme":"## Linear Attention Transformer\n\n\u003cimg src=\"./linear-attention.png\" width=\"700px\" /\u003e\n\n[![PyPI version](https://badge.fury.io/py/linear-attention-transformer.svg)](https://badge.fury.io/py/linear-attention-transformer)\n\nA fully featured Transformer that mixes (QKᵀ)V local attention with Q(KᵀV) global attention (scales linearly with respect to sequence length) for efficient long-range language modeling.\n\n## Install\n\n```bash\n$ pip install linear-attention-transformer\n```\n\n## Usage\n\nLanguage model\n\n```python\nimport torch\nfrom linear_attention_transformer import LinearAttentionTransformerLM\n\nmodel = LinearAttentionTransformerLM(\n    num_tokens = 20000,\n    dim = 512,\n    heads = 8,\n    depth = 1,\n    max_seq_len = 8192,\n    causal = True,                  # auto-regressive or not\n    ff_dropout = 0.1,               # dropout for feedforward\n    attn_layer_dropout = 0.1,       # dropout right after self-attention layer\n    attn_dropout = 0.1,             # dropout post-attention\n    emb_dim = 128,                  # embedding factorization, to save on memory\n    dim_head = 128,                 # be able to fix the dimension of each head, making it independent of the embedding dimension and the number of heads\n    blindspot_size = 64,            # this gives the q(kv) attention a blindspot of 64 tokens back in the causal case, but gives back an order of magnitude return in memory savings. should be paired with local attention of at least a window size of this setting. setting this to 1 will allow for full q(kv) attention of past\n    n_local_attn_heads = 4,         # number of local attention heads for (qk)v attention. this can be a tuple specifying the exact number of local attention heads at that depth\n    local_attn_window_size = 128,   # receptive field of the local attention\n    reversible = True,              # use reversible nets, from Reformer paper\n    ff_chunks = 2,                  # feedforward chunking, from Reformer paper\n    ff_glu = True,                  # use GLU variant for feedforward\n    attend_axially = False,         # will fold the sequence by the local attention window size, and do an extra strided attention followed by a feedforward with the cheap q(kv) attention\n    shift_tokens = True             # add single token shifting, for great improved convergence\n).cuda()\n\nx = torch.randint(0, 20000, (1, 8192)).cuda()\nmodel(x) # (1, 8192, 512)\n```\n\nTransformer\n\n```python\nimport torch\nfrom linear_attention_transformer import LinearAttentionTransformer\n\nmodel = LinearAttentionTransformer(\n    dim = 512,\n    heads = 8,\n    depth = 1,\n    max_seq_len = 8192,\n    n_local_attn_heads = 4\n).cuda()\n\nx = torch.randn(1, 8192, 512).cuda()\nmodel(x) # (1, 8192, 512)\n```\n\nEncoder / decoder\n\n```python\nimport torch\nfrom linear_attention_transformer import LinearAttentionTransformerLM\n\nenc = LinearAttentionTransformerLM(\n    num_tokens = 20000,\n    dim = 512,\n    heads = 8,\n    depth = 6,\n    max_seq_len = 4096,\n    reversible = True,\n    n_local_attn_heads = 4,\n    return_embeddings = True\n).cuda()\n\ndec = LinearAttentionTransformerLM(\n    num_tokens = 20000,\n    dim = 512,\n    heads = 8,\n    depth = 6,\n    causal = True,\n    max_seq_len = 4096,\n    reversible = True,\n    receives_context = True,\n    n_local_attn_heads = 4\n).cuda()\n\nsrc = torch.randint(0, 20000, (1, 4096)).cuda()\nsrc_mask = torch.ones_like(src).bool().cuda()\n\ntgt = torch.randint(0, 20000, (1, 4096)).cuda()\ntgt_mask = torch.ones_like(tgt).bool().cuda()\n\ncontext = enc(src, input_mask = src_mask)\nlogits = dec(tgt, context = context, input_mask = tgt_mask, context_mask = src_mask)\n```\n\n## Linformer\n\nLinformer is another variant of attention with linear complexity championed by Facebook AI. It only works with non-autoregressive models of a fixed sequence length. If your problem satisfies that criteria, you may choose to try it out.\n\n```python\nfrom linear_attention_transformer import LinearAttentionTransformerLM, LinformerSettings\n\nsettings = LinformerSettings(k = 256)\n\nenc = LinearAttentionTransformerLM(\n    num_tokens = 20000,\n    dim = 512,\n    heads = 8,\n    depth = 6,\n    max_seq_len = 4096,\n    linformer_settings = settings\n).cuda()\n```\n\nYou can also used Linformer for the contextual attention layer, if the contextual keys are of a fixed sequence length.\n\n```python\nfrom linear_attention_transformer import LinearAttentionTransformerLM, LinformerContextSettings\n\nsettings = LinformerContextSettings(\n  seq_len = 2048,\n  k = 256\n)\n\ndec = LinearAttentionTransformerLM(\n    num_tokens = 20000,\n    dim = 512,\n    heads = 8,\n    depth = 6,\n    max_seq_len = 4096,\n    causal = True,\n    context_linformer_settings = settings,\n    receives_context = True\n).cuda()\n```\n\n## Images\n\nThis repository also contains a concise implementation of this efficient attention for images\n\n```python\nimport torch\nfrom linear_attention_transformer.images import ImageLinearAttention\n\nattn =ImageLinearAttention(\n  chan = 32,\n  heads = 8,\n  key_dim = 64       # can be decreased to 32 for more memory savings\n)\n\nimg = torch.randn(1, 32, 256, 256)\nattn(img) # (1, 32, 256, 256)\n```\n\n## Citations\n\n```bibtex\n@inproceedings{katharopoulos-et-al-2020,\n  author    = {Katharopoulos, A. and Vyas, A. and Pappas, N. and Fleuret, F.},\n  title     = {Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention},\n  booktitle = {Proceedings of the International Conference on Machine Learning (ICML)},\n  year      = {2020},\n  url       = {https://arxiv.org/abs/2006.16236}\n}\n```\n\n```bibtex\n@article{shen2019efficient,\n  author    = {Zhuoran Shen and\n               Mingyuan Zhang and\n               Haiyu Zhao and\n               Shuai Yi and\n               Hongsheng Li},\n  title     = {Efficient Attention: Attention with Linear Complexities},\n  journal   = {CoRR},\n  volume    = {abs/1812.01243},\n  year      = {2018},\n  url       = {http://arxiv.org/abs/1812.01243}\n}\n```\n\n```bibtex\n@inproceedings{kitaev2020reformer,\n    title       = {Reformer: The Efficient Transformer},\n    author      = {Nikita Kitaev and Lukasz Kaiser and Anselm Levskaya},\n    booktitle   = {International Conference on Learning Representations},\n    year        = {2020},\n    url         = {https://openreview.net/forum?id=rkgNKkHtvB}\n}\n```\n\n```bibtex\n@misc{shazeer2020glu,\n    title   = {GLU Variants Improve Transformer},\n    author  = {Noam Shazeer},\n    year    = {2020},\n    url     = {https://arxiv.org/abs/2002.05202}\n}\n```\n\n```bibtex\n@misc{wang2020linformer,\n    title   = {Linformer: Self-Attention with Linear Complexity},\n    author  = {Sinong Wang and Belinda Z. Li and Madian Khabsa and Han Fang and Hao Ma},\n    year    = {2020},\n    eprint  = {2006.04768}\n}\n```\n\n```bibtex\n@misc{bhojanapalli2020lowrank,\n    title   = {Low-Rank Bottleneck in Multi-head Attention Models},\n    author  = {Srinadh Bhojanapalli and Chulhee Yun and Ankit Singh Rawat and Sashank J. Reddi and Sanjiv Kumar},\n    year    = {2020},\n    eprint  = {2002.07028}\n}\n```\n\n```bibtex\n@techreport{zhuiyiroformer,\n    title   = {RoFormer: Transformer with Rotary Position Embeddings - ZhuiyiAI},\n    author  = {Jianlin Su},\n    year    = {2021},\n    url     = \"https://github.com/ZhuiyiTechnology/roformer\",\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Flinear-attention-transformer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flucidrains%2Flinear-attention-transformer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Flinear-attention-transformer/lists"}