{"id":22066384,"url":"https://github.com/jaketae/fnet","last_synced_at":"2025-05-13T02:08:13.844Z","repository":{"id":41256221,"uuid":"368147436","full_name":"jaketae/fnet","owner":"jaketae","description":"PyTorch implementation of FNet: Mixing Tokens with Fourier transforms","archived":false,"fork":false,"pushed_at":"2021-05-17T15:14:08.000Z","size":7,"stargazers_count":27,"open_issues_count":1,"forks_count":7,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-05-13T02:07:58.967Z","etag":null,"topics":["bert","fourier-transform","natural-language-processing","pytorch","transformer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jaketae.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-05-17T10:36:21.000Z","updated_at":"2025-04-23T09:31:16.000Z","dependencies_parsed_at":"2022-09-10T06:22:00.004Z","dependency_job_id":null,"html_url":"https://github.com/jaketae/fnet","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaketae%2Ffnet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaketae%2Ffnet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaketae%2Ffnet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaketae%2Ffnet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jaketae","download_url":"https://codeload.github.com/jaketae/fnet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253856656,"owners_count":21974582,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert","fourier-transform","natural-language-processing","pytorch","transformer"],"created_at":"2024-11-30T19:27:55.907Z","updated_at":"2025-05-13T02:08:13.824Z","avatar_url":"https://github.com/jaketae.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# FNet\n\nPyTorch implementation of [FNet: Mixing Tokens with Fourier Transforms](https://arxiv.org/abs/2105.03824v1).\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://miro.medium.com/max/1551/0*LE7Bqa1C-JIAWP7Z.png\"\u003e\n\u003c/p\u003e\n\n## Quickstart\n\nClone this repository.\n\n```\ngit clone https://github.com/jaketae/fnet.git\n```\n\nNavigate to the cloned directory. You can start using the model via\n\n```python\n\u003e\u003e\u003e from fnet import FNet\n\u003e\u003e\u003e model = FNet()\n```\n\nBy default, the model comes with the following parameters:\n\n```python\nFNet(\n    d_model=256,\n    expansion_factor=2,\n    dropout=0.5,\n    num_layers=6,\n)\n```\n\n## Summary\n\nWhile transformers have proven to be successful in various domains, its `O(n^2)` computation complexity has been considered a structural weakness. Many attempts have been made to optimize the model architecture. The authors of the paper present FNet, a model that replaces self-attention with standard unparametrized Fourier transforms. Not only is FNet faster and computationaly more efficient than the classic transformer, but it also retains 92% of BERT's accuracy on the GLUE benchmark. Given a smaller number of parameters, FNet outperformed transformers.\n\n## Resources\n\n-   [Original Paper](https://arxiv.org/abs/2105.03824v1)\n-   [Rishikesh's implementation](https://github.com/rishikksh20/FNet-pytorch)\n-   Image from [SyncedReview's Medium article](https://medium.com/syncedreview/google-replaces-bert-self-attention-with-fourier-transform-92-accuracy-7-times-faster-on-gpus-7a78e3e4ac0e)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaketae%2Ffnet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjaketae%2Ffnet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaketae%2Ffnet/lists"}