{"id":15600989,"url":"https://github.com/lucidrains/multistream-transformers","last_synced_at":"2025-10-18T05:31:33.504Z","repository":{"id":48335387,"uuid":"390844262","full_name":"lucidrains/multistream-transformers","owner":"lucidrains","description":"Implementation of Multistream Transformers in Pytorch","archived":false,"fork":false,"pushed_at":"2021-07-31T21:36:19.000Z","size":35892,"stargazers_count":53,"open_issues_count":0,"forks_count":3,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-13T23:34:12.553Z","etag":null,"topics":["artificial-intelligence","deep-learning","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lucidrains.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-07-29T20:34:23.000Z","updated_at":"2024-10-16T02:19:23.000Z","dependencies_parsed_at":"2022-09-14T00:52:43.754Z","dependency_job_id":null,"html_url":"https://github.com/lucidrains/multistream-transformers","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fmultistream-transformers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fmultistream-transformers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fmultistream-transformers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fmultistream-transformers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lucidrains","download_url":"https://codeload.github.com/lucidrains/multistream-transformers/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246326759,"owners_count":20759437,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","deep-learning","transformers"],"created_at":"2024-10-03T02:11:06.679Z","updated_at":"2025-10-18T05:31:33.408Z","avatar_url":"https://github.com/lucidrains.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg src=\"./multistream.png\" width=\"300px\"\u003e\u003c/img\u003e\n\n## Multistream Transformers\n\nImplementation of \u003ca href=\"https://arxiv.org/abs/2107.10342\"\u003eMultistream Transformers\u003c/a\u003e in Pytorch.\n\nThis repository deviates slightly from the paper, where instead of using the skip connection across all streams, it uses attention pooling across all tokens in the same position. This has produced the best results in my experiments with number of streams greater than 2.\n\n## Install\n\n```\n$ pip install multistream-transformers\n```\n\n## Usage\n\n```python\nimport torch\nfrom multistream_transformers import MultistreamTransformer\n\nmodel = MultistreamTransformer(\n    num_tokens = 256,         # number of tokens\n    dim = 512,                # dimension\n    depth = 4,                # depth\n    causal = True,            # autoregressive or not\n    max_seq_len = 1024,       # maximum sequence length\n    num_streams = 2           # number of streams - 1 would make it a regular transformer\n)\n\nx = torch.randint(0, 256, (2, 1024))\nmask = torch.ones((2, 1024)).bool()\n\nlogits = model(x, mask = mask) # (2, 1024, 256)\n```\n\n## Citations\n\n```bibtex\n@misc{burtsev2021multistream,\n    title   = {Multi-Stream Transformers}, \n    author  = {Mikhail Burtsev and Anna Rumshisky},\n    year    = {2021},\n    eprint  = {2107.10342},\n    archivePrefix = {arXiv},\n    primaryClass = {cs.CL}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Fmultistream-transformers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flucidrains%2Fmultistream-transformers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Fmultistream-transformers/lists"}