{"id":15601000,"url":"https://github.com/lucidrains/perceiver-ar-pytorch","last_synced_at":"2025-07-15T21:35:46.078Z","repository":{"id":37599679,"uuid":"504887693","full_name":"lucidrains/perceiver-ar-pytorch","owner":"lucidrains","description":"Implementation of Perceiver AR, Deepmind's new long-context attention network based on Perceiver architecture, in Pytorch","archived":false,"fork":false,"pushed_at":"2023-04-10T08:57:10.000Z","size":35814,"stargazers_count":88,"open_issues_count":5,"forks_count":4,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-07-11T13:07:18.911Z","etag":null,"topics":["artficial-intelligence","attention-mechanism","deep-learning","long-context","transformer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lucidrains.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-06-18T15:58:56.000Z","updated_at":"2025-06-03T07:43:42.000Z","dependencies_parsed_at":"2023-09-24T02:47:37.457Z","dependency_job_id":null,"html_url":"https://github.com/lucidrains/perceiver-ar-pytorch","commit_stats":{"total_commits":19,"total_committers":2,"mean_commits":9.5,"dds":"0.052631578947368474","last_synced_commit":"f6005174e73f18d12d260dc80b6c907d67556db9"},"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/lucidrains/perceiver-ar-pytorch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fperceiver-ar-pytorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fperceiver-ar-pytorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fperceiver-ar-pytorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fperceiver-ar-pytorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lucidrains","download_url":"https://codeload.github.com/lucidrains/perceiver-ar-pytorch/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fperceiver-ar-pytorch/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264958623,"owners_count":23689035,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artficial-intelligence","attention-mechanism","deep-learning","long-context","transformer"],"created_at":"2024-10-03T02:11:23.419Z","updated_at":"2025-07-15T21:35:46.015Z","avatar_url":"https://github.com/lucidrains.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg src=\"./perceiver-ar.png\" width=\"300px\"\u003e\u003c/img\u003e\n\n## Perceiver AR - Pytorch\n\nImplementation of \u003ca href=\"https://arxiv.org/abs/2202.07765\"\u003ePerceiver AR\u003c/a\u003e, Deepmind's new long-context attention network based on Perceiver architecture, in Pytorch.\n\n\u003ca href=\"https://storage.googleapis.com/perceiver-ar/index.html\"\u003eGenerated piano samples\u003c/a\u003e\n\nI am building this out of popular demand, not because I believe in the architecture. As someone else puts it succinctly, this is equivalent to an encoder / decoder transformer architecture where the encoder has 0 layers (and the decoder cross attention is restricted to 1 layer)\n\nHowever, the experimental results they provided are still worthwhile and I'll build it out so students and researchers alike can explore along this avenue.\n\n\u003ca href=\"https://github.com/google-research/perceiver-ar\"\u003eOfficial Jax repository\u003c/a\u003e\n\nUpdate: seems to be performing decently well on enwik8 with 4096 context length. maybe I was wrong to be pessimistic\n\n## Install\n\n```bash\n$ pip install perceiver-ar-pytorch\n```\n\n## Usage\n\n```python\nimport torch\nfrom perceiver_ar_pytorch import PerceiverAR\n\nmodel = PerceiverAR(\n    num_tokens = 20000,             # number of tokens\n    dim = 512,                      # model dimensions\n    depth = 8,                      # model depth\n    dim_head = 64,                  # attention head dimension\n    heads = 8,                      # attention heads\n    max_seq_len = 4096,             # total max sequence length\n    cross_attn_seq_len = 3072,      # the sequence length in which to attend to, but does not undergo self attention (must be less than max_seq_len)\n    cross_attn_dropout = 0.5,       # what percentage of the prefix to dropout during training, in paper they had extensive experimentation to show up to 50% dropout helped prevent overfitting\n)\n\nx = torch.randint(0, 20000, (1, 4096))\n\nlogits = model(x) # (1, 1024, 20000) - (4096 [seq len] - 3072 [perceived prefix] == 1024)\n```\n\n## Test\n\nEnwik8 at 4096\n\n```bash\n$ python train.py\n```\n\n## Citations\n\n```bibtex\n@article{Hawthorne2022GeneralpurposeLA,\n    title   = {General-purpose, long-context autoregressive modeling with Perceiver AR},\n    author  = {Curtis Hawthorne and Andrew Jaegle and Cătălina Cangea and Sebastian Borgeaud and Charlie Nash and Mateusz Malinowski and Sander Dieleman and Oriol Vinyals and Matthew M. Botvinick and Ian Simon and Hannah R. Sheahan and Neil Zeghidour and Jean-Baptiste Alayrac and Jo{\\~a}o Carreira and Jesse Engel},\n    journal = {ArXiv},\n    year    = {2022},\n    volume  = {abs/2202.07765}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Fperceiver-ar-pytorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flucidrains%2Fperceiver-ar-pytorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Fperceiver-ar-pytorch/lists"}