{"id":13737977,"url":"https://github.com/idiap/fast-transformers","last_synced_at":"2025-05-15T00:07:05.071Z","repository":{"id":40681416,"uuid":"272430903","full_name":"idiap/fast-transformers","owner":"idiap","description":"Pytorch library for fast transformer implementations","archived":false,"fork":false,"pushed_at":"2023-03-23T16:46:47.000Z","size":361,"stargazers_count":1695,"open_issues_count":34,"forks_count":185,"subscribers_count":24,"default_branch":"master","last_synced_at":"2025-04-13T19:49:52.813Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/idiap.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-06-15T12:19:51.000Z","updated_at":"2025-04-11T14:26:17.000Z","dependencies_parsed_at":"2024-01-07T20:10:55.576Z","dependency_job_id":null,"html_url":"https://github.com/idiap/fast-transformers","commit_stats":{"total_commits":151,"total_committers":10,"mean_commits":15.1,"dds":0.2715231788079471,"last_synced_commit":"f22c13716fc748bb21a7b226ada7f7b5f87f867f"},"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idiap%2Ffast-transformers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idiap%2Ffast-transformers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idiap%2Ffast-transformers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idiap%2Ffast-transformers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/idiap","download_url":"https://codeload.github.com/idiap/fast-transformers/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254249198,"owners_count":22039029,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T03:02:07.850Z","updated_at":"2025-05-15T00:07:05.030Z","avatar_url":"https://github.com/idiap.png","language":"Python","funding_links":[],"categories":["Python","Transformer库与优化"],"sub_categories":[],"readme":"Fast Transformers\n=================\n\nTransformers are very successful models that achieve state of the art\nperformance in many natural language tasks. However, it is very difficult to\nscale them to long sequences due to the quadratic scaling of self-attention.\n\nThis library was developed for our research on fast attention for transformers.\nYou can find a list of our papers `in the docs\n\u003chttps://fast-transformers.github.io\u003e`_ as well as related papers and papers\nthat we have implemented.\n\nQuick-start\n-----------\n\nThe following code builds a transformer with softmax attention and one with\nlinear attention and compares the time required by each to encode a sequence\nwith 1000 elements.\n\n.. code:: python\n\n    import torch\n    from fast_transformers.builders import TransformerEncoderBuilder\n\n    # Create the builder for our transformers\n    builder = TransformerEncoderBuilder.from_kwargs(\n        n_layers=8,\n        n_heads=8,\n        query_dimensions=64,\n        value_dimensions=64,\n        feed_forward_dimensions=1024\n    )\n\n    # Build a transformer with softmax attention\n    builder.attention_type = \"full\"\n    softmax_model = builder.get()\n\n    # Build a transformer with linear attention\n    builder.attention_type = \"linear\"\n    linear_model = builder.get()\n\n    # Construct the dummy input\n    X = torch.rand(10, 1000, 8*64)\n\n    # Prepare everythin for CUDA\n    X = X.cuda()\n    softmax_model.cuda()\n    softmax_model.eval()\n    linear_model.cuda()\n    linear_model.eval()\n\n    # Warmup the GPU\n    with torch.no_grad():\n        softmax_model(X)\n        linear_model(X)\n    torch.cuda.synchronize()\n\n    # Measure the execution time\n    softmax_start = torch.cuda.Event(enable_timing=True)\n    softmax_end = torch.cuda.Event(enable_timing=True)\n    linear_start = torch.cuda.Event(enable_timing=True)\n    linear_end = torch.cuda.Event(enable_timing=True)\n\n    with torch.no_grad():\n        softmax_start.record()\n        y = softmax_model(X)\n        softmax_end.record()\n        torch.cuda.synchronize()\n        print(\"Softmax: \", softmax_start.elapsed_time(softmax_end), \"ms\")\n        # Softmax: 144 ms (on a GTX1080Ti)\n\n    with torch.no_grad():\n        linear_start.record()\n        y = linear_model(X)\n        linear_end.record()\n        torch.cuda.synchronize()\n        print(\"Linear: \", linear_start.elapsed_time(linear_end), \"ms\")\n        # Linear: 68 ms (on a GTX1080Ti)\n\nDependencies \u0026 Installation\n---------------------------\n\nThe fast transformers library has the following dependencies:\n\n* PyTorch\n* C++ toolchain\n* CUDA toolchain (if you want to compile for GPUs)\n\nFor most machines installation should be as simple as:\n\n.. code:: bash\n\n    pip install --user pytorch-fast-transformers\n\nNote: macOS users should ensure they have `llvm` and `libomp` installed.\nUsing the `homebrew \u003chttps://brew.sh\u003e`_ package manager, this can be\naccomplished by running `brew install llvm libomp`.\n\nDocumentation\n-------------\n\nThere exists a dedicated `documentation site\n\u003chttps://fast-transformers.github.io/\u003e`_ but you are also encouraged to read\nthe `source code \u003chttps://github.com/idiap/fast-transformers\u003e`_.\n\nResearch\n--------\n\nOurs\n~~~~\n\nTo read about the theory behind some attention implementations in this library\nwe encourage you to follow our research.\n\n* Transformers are RNNs: Fast Autoregressive Transformers with\n  Linear Attention (`2006.16236 \u003chttps://arxiv.org/abs/2006.16236\u003e`_)\n* Fast Transformers with Clustered Attention\n  (`2007.04825 \u003chttps://arxiv.org/abs/2007.04825\u003e`_)\n\nIf you found our research helpful or influential please consider citing\n\n.. code::\n\n    @inproceedings{katharopoulos_et_al_2020,\n        author = {Katharopoulos, A. and Vyas, A. and Pappas, N. and Fleuret, F.},\n        title = {Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention},\n        booktitle = {Proceedings of the International Conference on Machine Learning (ICML)},\n        year = {2020}\n    }\n\n    @article{vyas_et_al_2020,\n        author={Vyas, A. and Katharopoulos, A. and Fleuret, F.},\n        title={Fast Transformers with Clustered Attention},\n        booktitle = {Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS)},\n        year={2020}\n    }\n\nBy others\n~~~~~~~~~\n\n* Efficient Attention: Attention with Linear Complexities (`1812.01243\n  \u003chttps://arxiv.org/abs/1812.01243\u003e`_)\n* Linformer: Self-Attention with Linear Complexity (`2006.04768\n  \u003chttps://arxiv.org/abs/2006.04768\u003e`_)\n* Reformer: The Efficient Transformer (`2001.04451\n  \u003chttps://arxiv.org/abs/2001.04451\u003e`_)\n\nSupport, License and Copyright\n------------------------------\n\nThis software is distributed with the **MIT** license which pretty much means that\nyou can use it however you want and for whatever reason you want. All the\ninformation regarding support, copyright and the license can be found in the\n`LICENSE \u003chttps://github.com/idiap/fast-transformers/blob/master/LICENSE\u003e`_\nfile in the repository.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidiap%2Ffast-transformers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fidiap%2Ffast-transformers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidiap%2Ffast-transformers/lists"}