{"id":13752843,"url":"https://github.com/mlpen/Nystromformer","last_synced_at":"2025-05-09T20:34:30.927Z","repository":{"id":49821719,"uuid":"320443850","full_name":"mlpen/Nystromformer","owner":"mlpen","description":null,"archived":false,"fork":false,"pushed_at":"2023-10-18T06:50:45.000Z","size":11100,"stargazers_count":364,"open_issues_count":9,"forks_count":41,"subscribers_count":9,"default_branch":"main","last_synced_at":"2024-11-16T05:32:21.980Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mlpen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-12-11T02:20:47.000Z","updated_at":"2024-11-13T04:14:42.000Z","dependencies_parsed_at":"2024-08-03T09:14:45.300Z","dependency_job_id":null,"html_url":"https://github.com/mlpen/Nystromformer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlpen%2FNystromformer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlpen%2FNystromformer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlpen%2FNystromformer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlpen%2FNystromformer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mlpen","download_url":"https://codeload.github.com/mlpen/Nystromformer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253321780,"owners_count":21890464,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T09:01:11.639Z","updated_at":"2025-05-09T20:34:25.884Z","avatar_url":"https://github.com/mlpen.png","language":"Python","funding_links":[],"categories":["Transformer库与优化"],"sub_categories":[],"readme":"# Nystromformer: A Nystrom-based Algorithm for Approximating Self-Attention\n\n### January 11, 2022\nNystromformer has been added to huggingface by @novice03. huggingface documentation can be found [here](https://huggingface.co/docs/transformers/master/model_doc/nystromformer).\n\n### April 6, 2021\nWe extended segment-means to compute landmarks without requiring the sequence length divisible by the number of landmarks. Then we used this Nystromformer to \nperform deployment of T2T-Vit_t-14 for image classification without retraining. Our T2T-ViT-Nys-14 achieves 78% top-1 accuracy, outperforming performer/Linformer +4.3%/+12.7% for the direct deployment. \n\n### Feb 27th, 2021\nWe fixed the coefficient computation of initial Z_0, which can lead to faster convergence to pseudoinverse. The original implementation has a scale difference. \nWe leave the original as a default option. The added initialization is recommended. Thanks @sbodenstein for pointing out the difference. \n\n### Feb 17th, 2021\nWe have released the source code of PyTorch reimplementation of Long Range Arena (LRA) benchmark, which is to evaluate the generalization ability of models on diverse longer sequence tasks. Our codes are based on the [official Jax LRA implementation](https://github.com/google-research/long-range-arena). Reformer PyTorch implementation is from [huggingface](https://github.com/huggingface/transformers) and Performer PyTorch implementation is from [lucidrains](https://github.com/lucidrains/performer-pytorch).\n\n### Feb 14th, 2021\nWe have released the scores on individual LRA tasks. \n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"img/LRA.png\"\u003e\n\u003c/p\u003e\n\n### Feb 9th, 2021\nWe have release the average score across LRA tasks.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"img/avg_LRA.png\"\u003e\n\u003c/p\u003e\n\n\nTransformers have emerged as a powerful workhorse for a broad range of natural language processing tasks. A key component that drives the impressive performance of Transformers is their self-attention mechanism that identifies/encodes the influence or dependence of other tokens for each specific token. Its benefits notwithstanding, the quadratic complexity of self-attention on the input sequence length has limited its application to longer sequences – a topic being actively studied in the community. To address this limitation, we propose Nystromformer – a model that exhibits excellent scalability as a function of sequence length. Our idea is based on adapting the Nystrom method to approximate the standard self-attention with an efficient O(n) complexity.\n\n## Requirements\n\n```\ndocker, nvidia-docker\n```\n\n## Datasets\n\nThe pretraining dataset consists of English Wikipedia and BookCorpus. For pretraining on long sequence, we added one third Stories and one third Realnews. All downloaded data files should be placed in the corresponding folder under `data-preprocessing`. The original format of English Wikipedia dump is preprocessed using\n[wikiextractor](https://github.com/attardi/wikiextractor), and the resulting files are placed in `data-preprocessing/wiki`. Then, run `data-preprocessing/\u003cdataset\u003e/preprocess.py` under each corresponding folder to generate data files of unified format. After preprocessing, run `data-preprocessing/preprocess_data_\u003clength\u003e.py` to generate pretraining data of specific sequence length.\n\n## Pretraining\n\nTo start pretraining of a specific configuration: create a folder `\u003cmodel\u003e` (for example, `nystrom-512`) and write `\u003cmodel\u003e/config.json` to specify model and training configuration, then under `\u003cmodel\u003e` folder, run\n```\ndocker run --rm --name=pretrain \\\n  --network=host --ipc=host --gpus all \\\n  -v \"$PWD/../data-preprocessing/512-roberta:/dataset\" \\\n  -v \"$PWD/../code:/code\" \\\n  -v \"$PWD:/model\" \\\n  -d mlpen/bert_env:0 \\\n  /bin/bash -c \\\n  \"python3 /code/run_pretrain.py \u003e\u003e /model/pretrain.txt 2\u003e\u00261\"\n```\nAll outputs will be redirected to `\u003cmodel\u003e/pretrain.txt`. The command will create a `\u003cmodel\u003e/model` folder holding all checkpoints and log file. The training can be stopped anytime by running `docker kill pretrain`, and can be resumed from the last checkpoint using the same command for starting pretraining.\n\n## Pretraining from Different Model's Checkpoint\n\nCopy a checkpoint (one of `.model` or `.cp` file) from `\u003cdiff_model\u003e/model` folder to `\u003cmodel\u003e` folder and add a key-value pair in `\u003cmodel\u003e/config.json`: `\"from_cp\": \"/model/\u003ccheckpoint_file\u003e\"`. One example is shown in `nystrom-4096/config.json`. This procedure also works for extending the max sequence length of a model (For example, use `nystrom-512` pretrained weights as initialization for `nystrom-4096`).\n\n## GLUE\n\nTo finetune model on GLUE tasks, download GLUE datasets and place them under `glue` folder, then under folder `\u003cmodel\u003e`, run\n```\ndocker run --rm --name=glue \\\n  --network=host --ipc=host --gpus all \\\n  -v \"$PWD/../glue:/glue\" \\\n  -v \"$PWD/../code:/code\" \\\n  -v \"$PWD:/model\" \\\n  -d mlpen/bert_env:0 \\\n  /bin/bash -c \\\n  \"python3 /code/run_glue.py --batch_size 32 --lr 3e-5 --epoch 5 --task MRPC --checkpoint 99 \u003e\u003e /model/glue.txt 2\u003e\u00261\"\n```\n`batch_size`, `lr`, `epoch`, `task`, `checkpoint` can be changed to finetune on different task, different hyperparameters, or different checkpoints. All outputs will be redirected to `\u003cmodel\u003e/glue.txt`. The log file is located at `\u003cmodel\u003e/model` folder.\n\n## Citation\n```\n@article{xiong2021nystromformer,\n  title={Nystr{\\\"o}mformer: A Nystr{\\\"o}m-based Algorithm for Approximating Self-Attention},\n  author={Xiong, Yunyang and Zeng, Zhanpeng and Chakraborty, Rudrasis and Tan, Mingxing and Fung, Glenn and Li, Yin and Singh, Vikas},\n  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},\n  year={2021}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmlpen%2FNystromformer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmlpen%2FNystromformer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmlpen%2FNystromformer/lists"}