{"id":19401188,"url":"https://github.com/google-research/long-range-arena","last_synced_at":"2026-05-12T17:30:19.218Z","repository":{"id":37754619,"uuid":"310891951","full_name":"google-research/long-range-arena","owner":"google-research","description":"Long Range Arena for Benchmarking Efficient Transformers","archived":false,"fork":false,"pushed_at":"2023-12-16T10:07:31.000Z","size":133,"stargazers_count":743,"open_issues_count":28,"forks_count":85,"subscribers_count":24,"default_branch":"main","last_synced_at":"2025-02-15T01:12:12.896Z","etag":null,"topics":["attention","deep-learning","flax","jax","nlp","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-11-07T17:01:07.000Z","updated_at":"2025-02-12T21:54:22.000Z","dependencies_parsed_at":"2024-11-24T13:01:50.556Z","dependency_job_id":"f5ebeb9d-7385-4b24-ad9f-59985d348f2c","html_url":"https://github.com/google-research/long-range-arena","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Flong-range-arena","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Flong-range-arena/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Flong-range-arena/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Flong-range-arena/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google-research","download_url":"https://codeload.github.com/google-research/long-range-arena/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240114297,"owners_count":19749836,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention","deep-learning","flax","jax","nlp","transformers"],"created_at":"2024-11-10T11:17:29.353Z","updated_at":"2026-05-12T17:30:19.150Z","avatar_url":"https://github.com/google-research.png","language":"Python","funding_links":[],"categories":["Datasets"],"sub_categories":["Applications"],"readme":"## Long-Range Arena (LRA: pronounced ELRA).\n\nLong-range arena is an effort toward systematic evaluation of efficient\ntransformer models. The project aims at establishing benchmark tasks/dtasets\nusing which we can evaluate transformer-based models in a systematic way, by\nassessing their generalization power, computational efficiency, memory\nfoot-print, etc.\n\nLong-range arena also implements different variants of Transformer models in\n[JAX](https://github.com/google/jax), using\n[Flax](https://github.com/google/flax).\n\nThis first initial release includes the benchmarks for the paper \"Long Range\nArena: A benchmark for Efficient Transformers.\n\nCurrently we have released all the necessary code to get started and run our\nbenchmarks on vanilla Transformers.\n\n## V2 release\n\n*Update* We have released the xformer models used in our experiments.\n\n\u003cs\u003e We are working on a 2nd update that will release more models and baselines for\n  this benchmark suite. Stay tuned. \u003c/s\u003e\n\nPlease see below for more examples on how to get started.\n\n\n\n#### Our experiments\n\nCurrent leaderboard results of all xformer results on our benchmark results. (as\nof 8th November 2020)\n\nModel           | ListOps   | Text      | Retrieval | Image     | Path      | Path-X | Avg\n--------------- | --------- | --------- | --------- | --------- | --------- | ------ | ---\nLocal Att       | 15.82     | 52.98     | 53.39     | 41.46     | 66.63     | FAIL   | 46.06\nLinear Trans.   | 16.13     | **65.90** | 53.09     | 42.34     | 75.30     | FAIL   | 50.55\nReformer        | **37.27** | 56.10     | 53.40     | 38.07     | 68.50     | FAIL   | 50.67\nSparse Trans.   | 17.07     | 63.58     | **59.59** | **44.24** | 71.71     | FAIL   | 51.24\nSinkhorn Trans. | 33.67     | 61.20     | 53.83     | 41.23     | 67.45     | FAIL   | 51.29\nLinformer       | 35.70     | 53.94     | 52.27     | 38.56     | 76.34     | FAIL   | 51.36\nPerformer       | 18.01     | 65.40     | 53.82     | 42.77     | **77.05** | FAIL   | 51.41\nSynthesizer     | 36.99     | 61.68     | 54.67     | 41.61     | 69.45     | FAIL   | 52.88\nLongformer      | 35.63     | 62.85     | 56.89     | 42.22     | 69.71     | FAIL   | 53.46\nTransformer     | 36.37     | 64.27     | 57.46     | 42.44     | 71.40     | FAIL   | 54.39\nBigBird         | 36.05     | 64.02     | 59.29     | 40.83     | 74.87     | FAIL   | **55.01**\n\n### Public External Entries\n\nWe list the entries of other papers and submissions that used our LRA benchmark.\n\n\nModel           | ListOps   | Text      | Retrieval | Image     | Path      | Path-X | Avg\n--------------- | --------- | --------- | --------- | --------- | --------- | ------ | ---\nIGLOO           | 39.23\t    |  82\t      |  75.5\t    |  47.0\t    | 67.50\t    | NA     |  62.25\nTLB             | 37.05     |  81.88    |  76.91    |  57.51    | 79.06     | FAIL   |  66.48\n\n\nIGLOO Submissions (by Vsevolod Sourkov) - https://github.com/redna11/lra-igloo \\\nTLB ([Temporal Latent Bottleneck](lra_benchmarks/models/transformer_tlb)) - [transformer_tlb](lra_benchmarks/models/transformer_tlb) \n\n## Citation\n\nIf you find out work useful, please cite our paper at:\n\n```\n@inproceedings{\ntay2021long,\ntitle={Long Range Arena : A Benchmark for Efficient Transformers },\nauthor={Yi Tay and Mostafa Dehghani and Samira Abnar and Yikang Shen and Dara Bahri and Philip Pham and Jinfeng Rao and Liu Yang and Sebastian Ruder and Donald Metzler},\nbooktitle={International Conference on Learning Representations},\nyear={2021},\nurl={https://openreview.net/forum?id=qVyeW-grC2k}\n}\n```\n\n**Note: Please also cite the original sources of these datasets! **\n\n## Adding results to the leaderboard.\n\nPlease send the link of the paper (arxiv, or published) to the Yi Tay or Mostafa\nDehghani (emails in paper) to include your new results to the leaderboard. Just\nlike above, we will add results to the external submission part of the leaderboard.\nThis is so that we do not encourage hill-climbing on the leaderboard but rather\nmeaningful side by side comparisons. \n\n## A note on evaluation and comparisons\n\n### Meaningful Comparisons\n\nWe intend for your benchmark to act as a tool and suite for inspecting model\nbehaviour. As such, if you're running a new setup and you have tuned hparams,\ndo consider running all the other models.\n\n### Apples-to Apples setting\n\nThis setting is for folks who want to compare with our published results\n*directly*.\n\nThe default hyperparameter setup (each benchmark should have a config file now).\nYou are not allowed to change hyperparameters such as embedding size, hidden\ndimensions, number of layers of the new model.\n\nThe new model should be within at best 10% larger in terms of parameters\ncompared to the base Transformer model in the provided config file.\n\n### Free-for-all Setting\n\nYou are allowed to run any model size and change any hyperparameter of the\nmodel. However, in the end, you'll not be allowed to report results from *our*\nleaderboard because they are no longer comparable. You can choose to rerun\nmodels from our library in a comparable setting.\n\n## Adding benchmarks or models to this suite\n\nIf you develop or could benefit from an extensive array of xformer baselines,\nplease feel free to let us know if you're interested in building new benchmarks.\nWe welcome contributions for new or older models that are not covered in the\nexisting suite.\n\n## What if I find a better config for an existing model?\n\nIn this paper, we did not prioritize doing hparam sweeps. If you happen to find\nan implementation related issue or a better hparam that allows a model to do\nbetter on a certain task, do send a PR (or a new config file) and we will\nrun the model again internally and report new results for the existing model.\n\n## I have a new Xyzformer, how do we add this to the benchmark.\n\nThe official results are *only* for code that have been verified and run in\nour codebase. We report all external submissions as *external*. Either submit a PR,\nan email showing us how to run your model in our codebase and we will update the\nresults accordingly. (Note due to bandwidth constraints this process will take\na substantial amount of time). \n\n# Example Usage\n\nTo run a task, run the train.py file in the corresponding task directory.\n(please see how to obtain the data for certain tasks if applicable).\n\n```\nPYTHONPATH=\"$(pwd)\":\"$PYTHON_PATH\" python lra_benchmarks/listops/train.py \\\n      --config=lra_benchmarks/listops/configs/transformer_base.py \\\n      --model_dir=/tmp/listops \\\n      --task_name=basic \\\n      --data_dir=$HOME/lra_data/listops/\n```\n\n## Dataset Setup\n\nThis section describes the methods to obtain the datasets and run the tasks in\nLRA.\n\nTo download the datasets, please download it from\n`gs://long-range-arena/lra_release`. If permissions fail, you may download the\nentire gziped file at\nhttps://storage.googleapis.com/long-range-arena/lra_release.gz.\n\n### ListOps\n\nThis task can be found at `/listops`. The datasets used in our experiments can\nbe found at these google cloud buckets and are in TSV format.\n\nIf you would like to go to longer/shorter sequence lengths, we also support\ngenerating your own split, run the following comment:\n\n```\nPYTHONPATH=\"$(pwd)\":\"$PYTHON_PATH\" python lra_benchmarks/data/listops.py -- \\\n  --output_dir=$HOME/lra_data/listops/\n```\n\n### Text Classification\n\nThis task can be found at `/text_classification`. No action is required because\nthis task is already found in tensorflow datasets. The code should run as it is.\n\n### Document Retrieval\n\nPlease download the dataset at (http://aan.how/download/). Please download the\ntrain/test/dev splits from our google cloud bucket. Unfortunately, we were not\nable to re-distribute this datasets and are only releasing the ids in the format\n`label paper1_id paper2_id`. You may download the data from the original source\nand extract the textual data.\n\n## Pixel-level Image Classification\n\nThis task can be found at `/image`. No action is required because this task is\nalready found in tensorflow datasets. It should work out of the box.\n\n## Pathfinder\n\nPlease see the `./data` directory, where the TFDS builder for the pathfinder\ndataset can be found. We generated different datasets for pathfinder task, with\ndifferent levels of difficulty using the script provided\n[here](https://github.com/drewlinsley/pathfinder). You can find information\nabout the parameters used for generatinng the data in the TFDS builder code in\n`./data/pathfinder`. We are preparing the exact data splits for release at the\nmoment.\n\n## Disclaimer\n\nThis is not an official Google product.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Flong-range-arena","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle-research%2Flong-range-arena","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Flong-range-arena/lists"}