{"id":13671374,"url":"https://github.com/thuml/Flowformer","last_synced_at":"2025-04-27T18:31:11.372Z","repository":{"id":37343808,"uuid":"500759528","full_name":"thuml/Flowformer","owner":"thuml","description":"About Code release for \"Flowformer: Linearizing Transformers with Conservation Flows\" (ICML 2022), https://arxiv.org/pdf/2202.06258.pdf","archived":false,"fork":false,"pushed_at":"2024-07-18T02:34:14.000Z","size":6670,"stargazers_count":298,"open_issues_count":1,"forks_count":28,"subscribers_count":9,"default_branch":"main","last_synced_at":"2024-08-03T09:09:30.499Z","etag":null,"topics":["deep-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thuml.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-06-07T08:45:10.000Z","updated_at":"2024-08-02T07:10:17.000Z","dependencies_parsed_at":"2022-07-14T07:00:40.582Z","dependency_job_id":"43249116-4266-4fe2-be82-57e3cafd59a0","html_url":"https://github.com/thuml/Flowformer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thuml%2FFlowformer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thuml%2FFlowformer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thuml%2FFlowformer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thuml%2FFlowformer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thuml","download_url":"https://codeload.github.com/thuml/Flowformer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224079330,"owners_count":17252267,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning"],"created_at":"2024-08-02T09:01:07.896Z","updated_at":"2024-11-11T09:30:21.605Z","avatar_url":"https://github.com/thuml.png","language":"Python","funding_links":[],"categories":["Python","Transformer库与优化"],"sub_categories":[],"readme":"# Flowformer (ICML 2022)\nFlowformer: Linearizing Transformers with Conservation Flows\n\n:triangular_flag_on_post:**News** (2024.07) [Mobile-Attention](https://openreview.net/pdf?id=VHtIDVaOKC), a mobile-device-tailored version of Flowformer, has been published in ICML 2024. The attention code can be found [here](https://github.com/thuml/MobileAttention/blob/main/Mobile_Attention.py). You can obtain a faster model by just replacing the canonical Attention Mechanism with our Mobile-Attention.\n\nTransformers have achieved impressive success in various areas. However, the attention mechanism has quadratic complexity, significantly impeding Transformers from dealing with numerous tokens and scaling up to bigger models. In pursuing the **linear complexity** and **task-universal** foundation model, we propose Flowformer [[paper]](https://arxiv.org/pdf/2202.06258.pdf) with the following merits:\n\n- **Linear complexity** w.r.t sequence length, can handle extremely long sequences (over 4k tokens)\n- **Without specific inductive bias**, purely derived from the flow network theory\n- **Task-universal**, showing strong performance in **$\\color{red}{\\text{Long sequence, Vision, NLP, Time series, RL}}$**.\n\n## Flow-Attention Design\n\nWe cast the attention mechanism into flow network, where the information flow is aggregated from the sources (values) to the sinks (results) through the learned flow capacities (attentions).\n\nBy conducting the conservation in both source and sink ascpects, we can bring competition into Flow-Attention design to avoid trivial attention in the spirit that \"fixed resource will cause competition''.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\".\\pic\\Flow-Attention.png\" height = \"250\" alt=\"\" align=center /\u003e\n\u003cbr\u003e\u003cbr\u003e\n\u003cb\u003eFigure 1.\u003c/b\u003e Flow-Attention with Competition and Allocation mechanisms.\n\u003c/p\u003e\n\n## Get Started\n\n1. Please refer to different folders for detailed experiment instructions.\n\n   Note: We have suffered a lot in configuring environments for different tasks. If you also have problems in solving the environment, feel free to contact us and discuss about it.\n\n2. List of benchmarks\n\n- [x] Core code: see `Flow_Attention.py`\n- [x] GPT-style Pytorch Module: see `Flowformer_TorchModule`\n- [x] Long Sequence Modeling in LRA: see `Flowformer_LRA`\n- [x] Vision Recognization in ImageNet-1K: see `Flowformer_CV`\n- [x] Language Modeling in WikiText-103: see `Flowformer_NLP`\n- [x] Time series classification  in UEA: see `Flowformer_TimeSeries`\n- [x] Reinforcement Learning in D4RL: see `Flowformer_RL`\n- [ ] CUDA speed up version\n\n## Main Results\n\nSee the [[paper]](https://arxiv.org/pdf/2202.06258.pdf) for detailed results, including **nearly 20 comparing baselines**. \n\n| Task                                  | Metrics                                          | Flowformer       | Performer    | Reformer     | Vanilla\u003cbr\u003eTransformer |\n| ------------------------------------- | ------------------------------------------------ | ---------------- | ------------ | ------------ | ---------------------- |\n| Long Sequence Modeling \u003cbr\u003e (LRA)     | Avg Acc (%) $\\uparrow$                           | **56.48**        | 51.41        | 50.67        | OOM                    |\n| Vision Recognization\u003cbr\u003e(ImageNet-1K) | Top-1 Acc (%) $\\uparrow$                         | **80.6**         | 78.1         | 79.6         | 78.7                   |\n| Language Modeling\u003cbr\u003e(WikiText-103)   | Perplexity $\\downarrow$                          | **30.8**         | 37.5         | 33.6         | 33.0                   |\n| Time series classification\u003cbr\u003e(UEA)   | Avg Acc (%) $\\uparrow$                           | **73.0**         | 71.5         | 71.9         | 71.9                   |\n| Offline RL\u003cbr\u003e(D4RL)                  | Avg Reward $\\uparrow$ \u003cbr\u003eAvg Deviation $\\downarrow$ | **73.5** $\\pm$ 2.9 | 63.8 $\\pm$ 7.6 | 63.9 $\\pm$ 2.9 | 72.2 $\\pm$ **2.6**       |\n\nVanilla Transformer means Decision Transorfomer in RL.\n\n## Attention Visualization\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\".\\pic\\Attention-visualization.png\" height = \"300\" alt=\"\" align=center /\u003e\n\u003cbr\u003e\u003cbr\u003e\n\u003cb\u003eFigure 2.\u003c/b\u003e Attention visualization. Flowformer can capture the essential parts successfully.\n\u003c/p\u003e\n\n## Citation\n\nIf you find this repo useful, please cite our paper. \n\n```\n@inproceedings{wu2022flowformer,\n  title={Flowformer: Linearizing Transformers with Conservation Flows},\n  author={Haixu Wu and Jialong Wu and Jiehui Xu and Jianmin Wang and Mingsheng Long},\n  booktitle={International Conference on Machine Learning},\n  year={2022}\n}\n```\n\n## Contact\n\nIf you have any questions or want to use the code, please contact wuhx23@mails.tsinghua.edu.cn.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthuml%2FFlowformer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthuml%2FFlowformer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthuml%2FFlowformer/lists"}