{"id":15601007,"url":"https://github.com/lucidrains/omninet-pytorch","last_synced_at":"2025-04-30T07:49:11.585Z","repository":{"id":57448599,"uuid":"343664586","full_name":"lucidrains/omninet-pytorch","owner":"lucidrains","description":"Implementation of OmniNet, Omnidirectional Representations from Transformers, in Pytorch","archived":false,"fork":false,"pushed_at":"2021-03-19T18:07:30.000Z","size":133,"stargazers_count":57,"open_issues_count":1,"forks_count":5,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-13T23:34:12.591Z","etag":null,"topics":["artificial-intelligence","attention-mechanism","deep-learning","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lucidrains.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-03-02T06:07:58.000Z","updated_at":"2024-12-09T09:04:51.000Z","dependencies_parsed_at":"2022-09-16T13:53:03.226Z","dependency_job_id":null,"html_url":"https://github.com/lucidrains/omninet-pytorch","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fomninet-pytorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fomninet-pytorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fomninet-pytorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fomninet-pytorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lucidrains","download_url":"https://codeload.github.com/lucidrains/omninet-pytorch/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246326759,"owners_count":20759437,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","attention-mechanism","deep-learning","transformers"],"created_at":"2024-10-03T02:11:34.285Z","updated_at":"2025-03-31T13:31:21.900Z","avatar_url":"https://github.com/lucidrains.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg src=\"./omninet.png\" width=\"400px\"\u003e\u003c/img\u003e\n\n## Omninet - Pytorch\n\nImplementation of \u003ca href=\"https://arxiv.org/abs/2103.01075\"\u003eOmniNet\u003c/a\u003e, Omnidirectional Representations from Transformers, in Pytorch. The authors propose that we should be attending to all the tokens of the previous layers, leveraging recent efficient attention advances to achieve this goal.\n\n## Install\n\n```bash\n$ pip install omninet-pytorch\n```\n\n## Usage\n\n```python\nimport torch\nfrom omninet_pytorch import Omninet\n\nomninet = Omninet(\n    dim = 512,                     # model dimension\n    depth = 6,                     # depth\n    dim_head = 64,                 # dimension per head\n    heads = 8,                     # number of heads\n    pool_layer_tokens_every = 3,   # key to this paper - every N layers, omni attend to all tokens of all layers\n    attn_dropout = 0.1,            # attention dropout\n    ff_dropout = 0.1,              # feedforward dropout\n    feature_redraw_interval = 1000 # how often to redraw the projection matrix for omni attention net - Performer\n)\n\nx = torch.randn(1, 1024, 512)\nmask = torch.ones(1, 1024).bool()\n\nomninet(x, mask = mask) # (1, 1024, 512)\n```\n\nCausal case, just use the class `OmninetCausal`. At the moment, it isn't faithful to the paper (I am using layer axial attention with layer positional embeddings to draw up information), but will fix this once I rework the linear attention CUDA kernel.\n\n```python\nimport torch\nfrom omninet_pytorch import OmninetCausal\n\nomninet = OmninetCausal(\n    dim = 512,                     # model dimension\n    depth = 6,                     # depth\n    dim_head = 64,                 # dimension per head\n    heads = 8,                     # number of heads\n    pool_layer_tokens_every = 3,   # key to this paper - every N layers, omni attend to all tokens of all layers\n    attn_dropout = 0.1,            # attention dropout\n    ff_dropout = 0.1               # feedforward dropout\n)\n\nx = torch.randn(1, 1024, 512)\nmask = torch.ones(1, 1024).bool()\n\nomninet(x, mask = mask) # (1, 1024, 512)\n```\n\n## Citations\n\n```bibtex\n@misc{tay2021omninet,\n    title   = {OmniNet: Omnidirectional Representations from Transformers}, \n    author  = {Yi Tay and Mostafa Dehghani and Vamsi Aribandi and Jai Gupta and Philip Pham and Zhen Qin and Dara Bahri and Da-Cheng Juan and Donald Metzler},\n    year    = {2021},\n    eprint  = {2103.01075},\n    archivePrefix = {arXiv},\n    primaryClass = {cs.CV}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Fomninet-pytorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flucidrains%2Fomninet-pytorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Fomninet-pytorch/lists"}