{"id":13752799,"url":"https://github.com/lucidrains/performer-pytorch","last_synced_at":"2025-05-15T09:07:41.024Z","repository":{"id":39092123,"uuid":"300789100","full_name":"lucidrains/performer-pytorch","owner":"lucidrains","description":"An implementation of Performer, a linear attention-based transformer, in Pytorch","archived":false,"fork":false,"pushed_at":"2022-02-02T20:33:32.000Z","size":35963,"stargazers_count":1130,"open_issues_count":44,"forks_count":146,"subscribers_count":17,"default_branch":"main","last_synced_at":"2025-05-15T09:07:30.859Z","etag":null,"topics":["artificial-intelligence","attention","attention-mechanism","deep-learning","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lucidrains.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-10-03T03:41:36.000Z","updated_at":"2025-05-14T07:54:15.000Z","dependencies_parsed_at":"2022-09-16T22:22:14.380Z","dependency_job_id":null,"html_url":"https://github.com/lucidrains/performer-pytorch","commit_stats":null,"previous_names":[],"tags_count":83,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fperformer-pytorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fperformer-pytorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fperformer-pytorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fperformer-pytorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lucidrains","download_url":"https://codeload.github.com/lucidrains/performer-pytorch/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254310515,"owners_count":22049469,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","attention","attention-mechanism","deep-learning","transformers"],"created_at":"2024-08-03T09:01:11.170Z","updated_at":"2025-05-15T09:07:36.016Z","avatar_url":"https://github.com/lucidrains.png","language":"Python","funding_links":[],"categories":["Transformer库与优化","Pytorch实用程序"],"sub_categories":[],"readme":"\u003cimg src=\"./favor+.png\" width=\"500px\"\u003e\u003c/img\u003e\n\n## Performer - Pytorch\n\n[![PyPI version](https://badge.fury.io/py/performer-pytorch.svg)](https://badge.fury.io/py/performer-pytorch)\n\nAn implementation of \u003ca href=\"https://arxiv.org/abs/2009.14794\"\u003ePerformer\u003c/a\u003e, a linear attention-based transformer variant with a **F**ast **A**ttention **V**ia positive **O**rthogonal **R**andom features approach (FAVOR+).\n\n## Install\n\n```bash\n$ pip install performer-pytorch\n```\n\nThen you must run the following, if you plan on training an autoregressive model\n\n```bash\n$ pip install -r requirements.txt\n```\n\n## Usage\n\nPerformer Language Model\n\n```python\nimport torch\nfrom performer_pytorch import PerformerLM\n\nmodel = PerformerLM(\n    num_tokens = 20000,\n    max_seq_len = 2048,             # max sequence length\n    dim = 512,                      # dimension\n    depth = 12,                     # layers\n    heads = 8,                      # heads\n    causal = False,                 # auto-regressive or not\n    nb_features = 256,              # number of random features, if not set, will default to (d * log(d)), where d is the dimension of each head\n    feature_redraw_interval = 1000, # how frequently to redraw the projection matrix, the more frequent, the slower the training\n    generalized_attention = False,  # defaults to softmax approximation, but can be set to True for generalized attention\n    kernel_fn = torch.nn.ReLU(),    # the kernel function to be used, if generalized attention is turned on, defaults to Relu\n    reversible = True,              # reversible layers, from Reformer paper\n    ff_chunks = 10,                 # chunk feedforward layer, from Reformer paper\n    use_scalenorm = False,          # use scale norm, from 'Transformers without Tears' paper\n    use_rezero = False,             # use rezero, from 'Rezero is all you need' paper\n    ff_glu = True,                  # use GLU variant for feedforward\n    emb_dropout = 0.1,              # embedding dropout\n    ff_dropout = 0.1,               # feedforward dropout\n    attn_dropout = 0.1,             # post-attn dropout\n    local_attn_heads = 4,           # 4 heads are local attention, 4 others are global performers\n    local_window_size = 256,        # window size of local attention\n    rotary_position_emb = True,     # use rotary positional embedding, which endows linear attention with relative positional encoding with no learned parameters. should always be turned on unless if you want to go back to old absolute positional encoding\n    shift_tokens = True             # shift tokens by 1 along sequence dimension before each block, for better convergence\n)\n\nx = torch.randint(0, 20000, (1, 2048))\nmask = torch.ones_like(x).bool()\n\nmodel(x, mask = mask) # (1, 2048, 20000)\n```\n\nPlain Performer, if you are working with say images or other modalities\n\n```python\nimport torch\nfrom performer_pytorch import Performer\n\nmodel = Performer(\n    dim = 512,\n    depth = 1,\n    heads = 8,\n    causal = True\n)\n\nx = torch.randn(1, 2048, 512)\nmodel(x) # (1, 2048, 512)\n```\n\nEncoder / Decoder - Made possible by \u003ca href=\"https://github.com/gulnazaki\"\u003eThomas Melistas\u003c/a\u003e\n\n```python\nimport torch\nfrom performer_pytorch import PerformerEncDec\n\nSRC_SEQ_LEN = 4096\nTGT_SEQ_LEN = 4096\nGENERATE_LEN = 512\n\nenc_dec = PerformerEncDec(\n    dim = 512,\n    tie_token_embed = True,\n    enc_num_tokens = 20000,\n    enc_depth = 6,\n    enc_heads = 8,\n    enc_max_seq_len = SRC_SEQ_LEN,\n    dec_num_tokens = 20000,\n    dec_depth = 6,\n    dec_heads = 8,\n    dec_max_seq_len = TGT_SEQ_LEN,\n)\n\nsrc = torch.randint(0, 20000, (1, SRC_SEQ_LEN))\ntgt = torch.randint(0, 20000, (1, TGT_SEQ_LEN))\nsrc_mask = torch.ones_like(src).bool()\ntgt_mask = torch.ones_like(src).bool()\n\n# train\nenc_dec.train()\nloss = enc_dec(src, tgt, enc_mask = src_mask, dec_mask = tgt_mask)\nloss.backward()\n\n# generate\ngenerate_in = torch.randint(0, 20000, (1, SRC_SEQ_LEN)).long()\ngenerate_out_prime = torch.tensor([[0.]]).long() # prime with \u003cbos\u003e token\nsamples = enc_dec.generate(generate_in, generate_out_prime, seq_len = GENERATE_LEN, eos_token = 1) # assume 1 is id of stop token\nprint(samples.shape) # (1, \u003c= GENERATE_LEN) decode the tokens\n```\n\n\nStandalone self-attention layer with linear complexity in respect to sequence length, for replacing trained full-attention transformer self-attention layers.\n\n```python\nimport torch\nfrom performer_pytorch import SelfAttention\n\nattn = SelfAttention(\n    dim = 512,\n    heads = 8,\n    causal = False,\n).cuda()\n\nx = torch.randn(1, 1024, 512).cuda()\nattn(x) # (1, 1024, 512)\n```\n\nCross attention is similarly\n\n```python\nimport torch\nfrom performer_pytorch import CrossAttention\n\nattn = CrossAttention(\n    dim = 512,\n    heads = 8\n).cuda()\n\nx = torch.randn(1, 1024, 512).cuda()\ncontext = torch.randn(1, 512, 512).cuda()\n\nattn(x, context = context) # (1, 1024, 512)\n```\n\nTo minimize model surgery, you could also simply rewrite the code, so that the attention step is done by the `FastAttention` module, as follows.\n\n```python\nimport torch\nfrom performer_pytorch import FastAttention\n\n# queries / keys / values with heads already split and transposed to first dimension\n# 8 heads, dimension of head is 64, sequence length of 512\nq = torch.randn(1, 8, 512, 64)\nk = torch.randn(1, 8, 512, 64)\nv = torch.randn(1, 8, 512, 64)\n\nattn_fn = FastAttention(\n    dim_heads = 64,\n    nb_features = 256,\n    causal = False\n)\n\nout = attn_fn(q, k, v) # (1, 8, 512, 64)\n# now merge heads and combine outputs with Wo\n```\n\n## Advanced\n\nAt the end of training, if you wish to fix the projection matrices to get the model to output deterministically, you can invoke the following\n\n```python\nmodel.fix_projection_matrices_()\n```\n\nNow your model will have fixed projection matrices across all layers\n\n## Citations\n\n```bibtex\n@misc{choromanski2020rethinking,\n    title   = {Rethinking Attention with Performers},\n    author  = {Krzysztof Choromanski and Valerii Likhosherstov and David Dohan and Xingyou Song and Andreea Gane and Tamas Sarlos and Peter Hawkins and Jared Davis and Afroz Mohiuddin and Lukasz Kaiser and David Belanger and Lucy Colwell and Adrian Weller},\n    year    = {2020},\n    eprint  = {2009.14794},\n    archivePrefix = {arXiv},\n    primaryClass = {cs.LG}\n}\n```\n\n```bibtex\n@inproceedings{kitaev2020reformer,\n    title       = {Reformer: The Efficient Transformer},\n    author      = {Nikita Kitaev and Lukasz Kaiser and Anselm Levskaya},\n    booktitle   = {International Conference on Learning Representations},\n    year        = {2020},\n    url         = {https://openreview.net/forum?id=rkgNKkHtvB}\n}\n```\n\n```bibtex\n@inproceedings{katharopoulos_et_al_2020,\n    author  = {Katharopoulos, A. and Vyas, A. and Pappas, N. and Fleuret, F.},\n    title   = {Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention},\n    booktitle = {Proceedings of the International Conference on Machine Learning (ICML)},\n    year    = {2020}\n}\n```\n\n```bibtex\n@misc{bachlechner2020rezero,\n    title   = {ReZero is All You Need: Fast Convergence at Large Depth},\n    author  = {Thomas Bachlechner and Bodhisattwa Prasad Majumder and Huanru Henry Mao and Garrison W. Cottrell and Julian McAuley},\n    year    = {2020},\n    url     = {https://arxiv.org/abs/2003.04887}\n}\n```\n\n```bibtex\n@article{1910.05895,\n    author  = {Toan Q. Nguyen and Julian Salazar},\n    title   = {Transformers without Tears: Improving the Normalization of Self-Attention},\n    year    = {2019},\n    eprint  = {arXiv:1910.05895},\n    doi     = {10.5281/zenodo.3525484},\n}\n```\n\n```bibtex\n@misc{shazeer2020glu,\n    title   = {GLU Variants Improve Transformer},\n    author  = {Noam Shazeer},\n    year    = {2020},\n    url     = {https://arxiv.org/abs/2002.05202}\n}\n```\n\n```bibtex\n@misc{roy*2020efficient,\n    title   = {Efficient Content-Based Sparse Attention with Routing Transformers},\n    author  = {Aurko Roy* and Mohammad Taghi Saffar* and David Grangier and Ashish Vaswani},\n    year    = {2020},\n    url     = {https://arxiv.org/pdf/2003.05997.pdf}\n}\n```\n\n```bibtex\n@misc{su2021roformer,\n    title   = {RoFormer: Enhanced Transformer with Rotary Position Embedding},\n    author  = {Jianlin Su and Yu Lu and Shengfeng Pan and Bo Wen and Yunfeng Liu},\n    year    = {2021},\n    eprint  = {2104.09864},\n    archivePrefix = {arXiv},\n    primaryClass = {cs.CL},\n    url     = {https://arxiv.org/abs/2104.09864}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Fperformer-pytorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flucidrains%2Fperformer-pytorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Fperformer-pytorch/lists"}