{"id":15600961,"url":"https://github.com/lucidrains/g-mlp-gpt","last_synced_at":"2025-04-14T12:34:03.177Z","repository":{"id":57435097,"uuid":"369059812","full_name":"lucidrains/g-mlp-gpt","owner":"lucidrains","description":"GPT, but made only out of MLPs","archived":false,"fork":false,"pushed_at":"2021-05-25T03:02:14.000Z","size":35750,"stargazers_count":88,"open_issues_count":0,"forks_count":9,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-03-28T01:50:37.995Z","etag":null,"topics":["artificial-intelligence","deep-learning","multilayer-perceptron"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lucidrains.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-05-20T02:45:34.000Z","updated_at":"2024-11-30T07:03:26.000Z","dependencies_parsed_at":"2022-09-04T15:33:34.017Z","dependency_job_id":null,"html_url":"https://github.com/lucidrains/g-mlp-gpt","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fg-mlp-gpt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fg-mlp-gpt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fg-mlp-gpt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fg-mlp-gpt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lucidrains","download_url":"https://codeload.github.com/lucidrains/g-mlp-gpt/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248881973,"owners_count":21176949,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","deep-learning","multilayer-perceptron"],"created_at":"2024-10-03T02:10:24.021Z","updated_at":"2025-04-14T12:34:03.149Z","avatar_url":"https://github.com/lucidrains.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## GPT - gMLP\n\nThis repository will attempt to crack long context autoregressive language modeling (GPT) using variations of \u003ca href=\"https://arxiv.org/abs/2105.08050\"\u003egMLPs\u003c/a\u003e. Specifically, it will contain a variant that does gMLP for local sliding windows. The hope is to be able to stretch a single GPU to be able to train context lengths of 4096 and above efficiently and well.\n\nYou can also add the \"tiny\" attention (as described in the paper) with the `attn_dim` keyword argument, which corresponds to the dimension of the single head (64 is recommended). You can pass in a tuple to customize different dimension per layer.\n\n## Install\n\n```bash\n$ pip install g-mlp-gpt\n```\n\n## Usage\n\n```python\nimport torch\nfrom g_mlp_gpt import gMLPGPT\n\nmodel = gMLPGPT(\n    num_tokens = 20000,\n    dim = 512,\n    depth = 4,\n    seq_len = 1024,\n    window = (128, 256, 512, 1024) # window sizes for each depth\n)\n\nx = torch.randint(0, 20000, (1, 1000))\nlogits = model(x) # (1, 1000, 20000)\n```\n\n16k context length\n\n```python\nimport torch\nfrom g_mlp_gpt import gMLPGPT\n\nmodel = gMLPGPT(\n    num_tokens = 20000,\n    dim = 512,\n    seq_len = 16384,\n    reversible = True,    # reversible networks\n    act = nn.Tanh(),      # tanh activation for spatial gating\n    depth = 12,\n    window = (\n        128,\n        128,\n        256,\n        512,\n        1024,\n        1024,\n        (2048, 2),    # window size of 2048, axial of 2\n        (2048, 2),\n        (4096, 4),\n        (4096, 4),\n        (8192, 8),    # window size of 8192, axial of 8\n        (8192, 8)\n    )\n).cuda()\n\nx = torch.randint(0, 20000, (1, 16384)).cuda()\nlogits = model(x) # (1, 16384, 20000)\n```\n\n## Citations\n\n```bibtex\n@misc{liu2021pay,\n    title   = {Pay Attention to MLPs}, \n    author  = {Hanxiao Liu and Zihang Dai and David R. So and Quoc V. Le},\n    year    = {2021},\n    eprint  = {2105.08050},\n    archivePrefix = {arXiv},\n    primaryClass = {cs.LG}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Fg-mlp-gpt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flucidrains%2Fg-mlp-gpt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Fg-mlp-gpt/lists"}