{"id":13753243,"url":"https://github.com/YerevaNN/warp","last_synced_at":"2025-05-09T20:35:07.327Z","repository":{"id":47466317,"uuid":"320386245","full_name":"YerevaNN/WARP","owner":"YerevaNN","description":"Code for ACL'2021 paper WARP 🌀 Word-level Adversarial ReProgramming. Outperforming `GPT-3` on SuperGLUE Few-Shot text classification. https://aclanthology.org/2021.acl-long.381/","archived":false,"fork":false,"pushed_at":"2021-10-03T19:31:03.000Z","size":87,"stargazers_count":83,"open_issues_count":3,"forks_count":16,"subscribers_count":8,"default_branch":"main","last_synced_at":"2024-11-16T05:32:38.860Z","etag":null,"topics":["adversarial","few-shot-learning","natural-language-processing","pretrained-models"],"latest_commit_sha":null,"homepage":"https://mahnerak.com/WARP","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/YerevaNN.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-12-10T20:47:59.000Z","updated_at":"2024-02-10T11:52:45.000Z","dependencies_parsed_at":"2022-08-20T13:31:02.995Z","dependency_job_id":null,"html_url":"https://github.com/YerevaNN/WARP","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YerevaNN%2FWARP","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YerevaNN%2FWARP/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YerevaNN%2FWARP/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YerevaNN%2FWARP/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/YerevaNN","download_url":"https://codeload.github.com/YerevaNN/WARP/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253321841,"owners_count":21890476,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adversarial","few-shot-learning","natural-language-processing","pretrained-models"],"created_at":"2024-08-03T09:01:18.932Z","updated_at":"2025-05-09T20:35:02.268Z","avatar_url":"https://github.com/YerevaNN.png","language":"Python","funding_links":[],"categories":["文本分类"],"sub_categories":[],"readme":"# 🌀 WARP: Word-level Adversarial ReProgramming\nThis repository contains code for ACL'2021 Paper [WARP: Word-level Adversarial ReProgramming](https://aclanthology.org/2021.acl-long.381/).\n\n\u003cimg src=\"https://svgshare.com/i/XpG.svg\"\u003e\n\u003csup\u003eWARP adds a few trainable embeddings around the input, which causes the masked language model to predict the sentiment of the sentence in the SST-2 task.\u003c/sup\u003e\n\nTransfer learning from pretrained language models recently became the dominant approach for solving many NLP tasks. A common approach to transfer learning for multiple tasks that maximize parameter sharing trains one or more task-specific layers on top of the language model.\n\nIn this paper, we present an alternative approach based on adversarial reprogramming, which extends earlier work on automatic prompt generation. Adversarial reprogramming attempts to learn task-specific word embeddings that, when concatenated to the input text, instruct the language model to solve the specified task.\n\nUsing up to 25K trainable parameters per task, this approach outperforms all existing methods that use up to 25M trainable parameters on the public leaderboard of the GLUE benchmark. Our method, initialized with task-specific human-readable prompts, also works in a few-shot setting, outperforming GPT-3 on two SuperGLUE tasks after training on just 32 samples.\n\n\n# Few-Shot Results\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003cth rowspan=\"2\"\u003eSet\u003c/th\u003e\n    \u003cth rowspan=\"2\"\u003eModel\u003c/th\u003e\n    \u003cth colspan=\"2\"\u003eCB\u003c/th\u003e\n    \u003cth\u003eRTE\u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"center\"\u003e\u003cb\u003eF\u003csub\u003e1\u003c/sub\u003e\u003c/b\u003e\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003cb\u003eAcc.\u003c/b\u003e\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003cb\u003eAcc.\u003c/b\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd rowspan=\"7\" align=\"center\"\u003edev\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eGPT-3 Small\u003c/td\u003e\n    \u003ctd align=\"right\"\u003e26.1\u003c/td\u003e\n    \u003ctd align=\"right\"\u003e42.9\u003c/td\u003e\n    \u003ctd align=\"right\"\u003e52.3\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eGPT-3 Med\u003c/td\u003e\n    \u003ctd align=\"right\"\u003e40.4\u003c/td\u003e\n    \u003ctd align=\"right\"\u003e58.9\u003c/td\u003e\n    \u003ctd align=\"right\"\u003e48.4\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eGPT-3\u003c/td\u003e\n    \u003ctd align=\"right\"\u003e57.2\u003c/td\u003e\n    \u003ctd align=\"right\"\u003e82.1\u003c/td\u003e\n    \u003ctd align=\"right\"\u003e72.9\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003ePET (ALBERT)\u003c/td\u003e\n    \u003ctd align=\"right\"\u003e59.4\u003c/td\u003e\n    \u003ctd align=\"right\"\u003e85.1\u003c/td\u003e\n    \u003ctd align=\"right\"\u003e69.8\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eiPET (ALBERT)\u003c/td\u003e\n    \u003ctd align=\"right\"\u003e92.4\u003c/td\u003e\n    \u003ctd align=\"right\"\u003e92.9\u003c/td\u003e\n    \u003ctd align=\"right\"\u003e74.0\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eWARP\u003csub\u003einit\u003c/sub\u003e (ALBERT) \u003c/td\u003e\n    \u003ctd align=\"right\"\u003e84.0\u003c/td\u003e\n    \u003ctd align=\"right\"\u003e87.5\u003c/td\u003e\n    \u003ctd align=\"right\"\u003e71.8\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd rowspan=\"6\" align=\"center\"\u003etest\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eGPT-3                             \u003c/td\u003e\n    \u003ctd align=\"right\"\u003e52.0\u003c/td\u003e\n    \u003ctd align=\"right\"\u003e75.6\u003c/td\u003e\n    \u003ctd align=\"right\"\u003e69.0\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003ePET (ALBERT)                      \u003c/td\u003e\n    \u003ctd align=\"right\"\u003e60.2\u003c/td\u003e\n    \u003ctd align=\"right\"\u003e87.2\u003c/td\u003e\n    \u003ctd align=\"right\"\u003e67.2\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eiPET (ALBERT)                     \u003c/td\u003e\n    \u003ctd align=\"right\"\u003e79.9\u003c/td\u003e\n    \u003ctd align=\"right\"\u003e88.8\u003c/td\u003e\n    \u003ctd align=\"right\"\u003e70.8\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eWARP\u003csub\u003einit\u003c/sub\u003e (ALBERT) \u003c/td\u003e\n    \u003ctd align=\"right\"\u003e70.2\u003c/td\u003e\n    \u003ctd align=\"right\"\u003e82.4\u003c/td\u003e\n    \u003ctd align=\"right\"\u003e69.1\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\u003csup\u003eResults on SuperGLUE benchmark. The results for the test set are obtained from SuperGLUE evaluation server.\nWe only show systems performing in a similar few-shot training setup using 32 examples.\u003c/sup\u003e\n\n# Setup\nThe code requires YerevaNN's internal version of `allennlp`\n```\ngit clone https://github.com/YerevaNN/allennlp\ngit checkout warp\npip install .\n```\n\n# Training\n\n\n### Linear Probing\n```sh\nfor DATASET in 'cola' 'sst2' 'mrpc' 'qqp' 'stsb' 'mnli' 'rte' 'wnli' 'qnli'\ndo\n    export HPARAMS='{\n        \"dataset\": \"'$DATASET'\",\n        \"lr\": 0.0001,\n        \"num_epochs\": 20,\n        \"prompts\": [],\n        \"reorder_optimized\": false,\n        \"max_batch_size\": 8,\n        \"max_tokens_sq\": 262144, \"on_logits\":  false, \"pooling_index\":  null, \"seed\":  1}'\n    python -m allennlp train \\\n    -s .aim/baseline-linear-${DATASET} configs/warp.jsonnet\ndone\n```\n\n### WARP_0\n```sh\nfor DATASET in 'cola' 'sst2' 'mrpc' 'qqp' 'stsb' 'mnli' 'rte' 'wnli' 'qnli'\ndo\n    export HPARAMS='{\n        \"dataset\": \"'$DATASET'\",\n        \"lr\": 0.0001,\n        \"num_epochs\": 20,\n        \"prompts\": [null, \"\u003cmask\u003e\"],\n        \"reorder_optimized\": true,\n        \"max_batch_size\": 8,\n        \"max_tokens_sq\": 262144,\n        \"on_logits\": \"pre_decoder_layer_norm\",\n        \"pooling_index\": 1,\n        \"seed\": 1\n    }'\n    python -m allennlp train \\\n    -s .aim/baseline-warp_0-${DATASET} configs/warp.jsonnet\ndone\n```\n\n## Training WARP\n\n```sh\nexport DATASET=\"rte\"\nexport HPARAMS='{\n    \"benchmark\":\"super_glue\",\n    \"classifier_init\":null,\n    \"dataset\":\"'$DATASET'\",\n    \"ensure_whitespace_between\":false,\n    \"lr\":0.001,\n    \"max_batch_size\":8,\n    \"max_tokens_sq\":262144,\n    \"num_epochs\":30,\n    \"prompt_better_init\":\"\u003cmask\u003e\",\n    \"prompts\":[-10,-11,-12,-13,-14,null,-15,-16,-17,-18,-19,\"\u003cmask\u003e\",-20,-21,-22,-23,-24,null,-25,-26,-27,-28,-29],\n    \"seed\":1,\n    \"transformer_model\":\"roberta-large\"\n}'\npython -m allennlp train \\\n-s .aim/t-${DATASET} configs/warp.jsonnet\n```\n\n## WARP_init\n## Few-Shot Experiments\n```sh\nexport HPARAMS='{\n    \"benchmark\":\"super_glue\",\n    \"classifier_init\": {\n        \"entailment\": \" yes\",\n        \"not_entailment\": \" instead\"\n    },\n    \"dataset\":\"few_rte\",\n    \"eval_mode\":false,\n    \"lr\":0.001,\n    \"max_batch_size\":2,\n    \"max_tokens_sq\":131072,\n    \"num_epochs\":100,\n    \"num_gradient_accumulation_steps\":2,\n    \"prompt_better_init\": \"[PAD]\",\n    \"prompts\":[-10,-11,[-14,\"\\\"\"],null,[-15,\"\\\"\"],  [-16, \"?\"], \"\u003cmask\u003e\", [-20, \",\"], null, [-29, \"!\"],-30,-31],\n    \"seed\":3,\n    \"str_cut_frac\":0,\n    \"transformer_model\":\"albert-xxlarge-v2\",\n    \"validation_metric\": null\n}'\npython -m allennlp train \\\n-s .aim/t-${DATASET}-`date +%s` configs/warp.jsonnet\n```\n\n\n```sh\nexport HPARAMS='{\n   \"benchmark\":\"super_glue\",\n   \"classifier_init\":{\n      \"entailment\":\" yes\",\n      \"not_entailment\":\" instead\"\n   },\n   \"dataset\":\"few_rte\",\n   \"grad_norm\":1,\n   \"lr\":0.001,\n   \"max_batch_size\":2,\n   \"max_tokens_sq\":131072,\n   \"num_epochs\":30,\n   \"num_gradient_accumulation_steps\":2,\n   \"prompt_better_init\":\"[PAD]\",\n   \"prompts\":[-10,-11,[-14,\"\\\"\"],null,[-15,\"\\\"\"],[-16,\"?\"],\"\u003cmask\u003e\",[-20,\",\"],null,[-29,\"!\"],-30,-31],\n   \"seed\":1,\n   \"str_cut_frac\":0.06,\n   \"transformer_model\":\"albert-xxlarge-v2\",\n   \"validation_metric\":\"+training_val_metric\"\n}'\npython -m allennlp train \\\n-s .aim/t-${DATASET}-`date +%s` configs/warp.jsonnet\n```\n\n## Evaluation\n\n```sh\npython -m allennlp predict \\\n  --silent --use-dataset-reader --cuda-device 0 \\\n  --batch-size 50 \\\n  --predictor glue --output-file v0.1/AX.tsv /data/arp/.aim/H-93ae5ae9 ax/test\n```\n\n```sh\npython -m allennlp predict \\\n  --silent --use-dataset-reader --cuda-device 0 \\\n  --batch-size 50 \\\n  --predictor glue --output-file v0.1/MNLI-m.tsv /data/arp/.aim/H-93ae5ae9 test_matched\n```\n\n## Citation\nIf you want to refer to our work use this bibTeX:\n```\n@inproceedings{hambardzumyan-etal-2021-warp,\n    title = \"{WARP}: {W}ord-level {A}dversarial {R}e{P}rogramming\",\n    author = \"Hambardzumyan, Karen  and\n      Khachatrian, Hrant  and\n      May, Jonathan\",\n    booktitle = \"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)\",\n    month = aug,\n    year = \"2021\",\n    address = \"Online\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2021.acl-long.381\",\n    doi = \"10.18653/v1/2021.acl-long.381\",\n    pages = \"4921--4933\"\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FYerevaNN%2Fwarp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FYerevaNN%2Fwarp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FYerevaNN%2Fwarp/lists"}