{"id":22066386,"url":"https://github.com/jaketae/g-mlp","last_synced_at":"2025-05-13T02:05:43.186Z","repository":{"id":49111109,"uuid":"368625951","full_name":"jaketae/g-mlp","owner":"jaketae","description":"PyTorch implementation of Pay Attention to MLPs","archived":false,"fork":false,"pushed_at":"2021-06-28T15:37:40.000Z","size":189,"stargazers_count":40,"open_issues_count":1,"forks_count":6,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-13T02:05:25.296Z","etag":null,"topics":["attention","image-classification","mlp","natural-language-processing","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jaketae.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-05-18T18:11:11.000Z","updated_at":"2024-12-20T05:28:16.000Z","dependencies_parsed_at":"2022-09-24T03:40:47.357Z","dependency_job_id":null,"html_url":"https://github.com/jaketae/g-mlp","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaketae%2Fg-mlp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaketae%2Fg-mlp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaketae%2Fg-mlp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaketae%2Fg-mlp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jaketae","download_url":"https://codeload.github.com/jaketae/g-mlp/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253856654,"owners_count":21974581,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention","image-classification","mlp","natural-language-processing","pytorch"],"created_at":"2024-11-30T19:27:56.169Z","updated_at":"2025-05-13T02:05:43.095Z","avatar_url":"https://github.com/jaketae.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# gMLP\n\nPyTorch implementation of [Pay Attention to MLPs](https://arxiv.org/abs/2105.08050).\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./figure.png\"\u003e\n\u003c/p\u003e\n\n## Quickstart\n\nClone this repository.\n\n```\ngit clone https://github.com/jaketae/g-mlp.git\n```\n\nNavigate to the cloned directory. You can use the barebone gMLP model via\n\n```python\n\u003e\u003e\u003e from g_mlp import gMLP\n\u003e\u003e\u003e model = gMLP()\n```\n\nBy default, the model comes with the following parameters:\n\n```python\ngMLP(\n    d_model=256,\n    d_ffn=512,\n    seq_len=256,\n    num_layers=6,\n)\n```\n\n## Usage\n\nThe repository also contains gMLP models specifically for language modeling and image classification.\n\n### NLP\n\n`gMLPForLanguageModeling` shares the same default parameters as `gMLP`, with `num_tokens=10000` as an added parameter that represents the size of the token embedding table.\n\n```python\n\u003e\u003e\u003e from g_mlp import gMLPForLanguageModeling\n\u003e\u003e\u003e model = gMLPForLanguageModeling()\n\u003e\u003e\u003e tokens = torch.randint(0, 10000, (8, 256))\n\u003e\u003e\u003e model(tokens).shape\ntorch.Size([8, 256, 256])\n```\n\n### Computer Vision\n\n`gMLPForImageClassification` is a ViT-esque version of gMLP that includes a patch creating layer and a final classification head.\n\n```python\n\u003e\u003e\u003e from g_mlp import gMLPForImageClassification\n\u003e\u003e\u003e model = gMLPForImageClassification()\n\u003e\u003e\u003e images = torch.randn(8, 3, 256, 256)\n\u003e\u003e\u003e model(images).shape\ntorch.Size([8, 1000])\n```\n\n## Summary\n\nThe authors of the paper present gMLP, an an attention-free all-MLP architecture based on spatial gating units. gMLP achieves parity with transformer models such as ViT and BERT on language and vision downstream tasks. The authors also show that gMLP scales with increased data and number of parameters, suggesting that self-attention is not a necessary component for designing performant models.\n\n## Resources\n\n-   [Original Paper](https://arxiv.org/abs/2105.08050)\n-   [Phil Wang's implementation](https://github.com/lucidrains/g-mlp-pytorch)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaketae%2Fg-mlp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjaketae%2Fg-mlp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaketae%2Fg-mlp/lists"}