{"id":13408896,"url":"https://github.com/THUDM/P-tuning-v2","last_synced_at":"2025-03-14T13:32:15.879Z","repository":{"id":37290795,"uuid":"417157956","full_name":"THUDM/P-tuning-v2","owner":"THUDM","description":"An optimized deep prompt tuning strategy comparable to fine-tuning across scales and tasks","archived":false,"fork":false,"pushed_at":"2023-11-16T04:38:09.000Z","size":1481,"stargazers_count":1974,"open_issues_count":34,"forks_count":201,"subscribers_count":29,"default_branch":"main","last_synced_at":"2024-10-21T17:19:04.598Z","etag":null,"topics":["natural-language-processing","p-tuning","parameter-efficient-learning","pretrained-language-model","prompt-tuning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/THUDM.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-14T14:16:05.000Z","updated_at":"2024-10-18T05:19:27.000Z","dependencies_parsed_at":"2024-01-14T08:55:22.064Z","dependency_job_id":"e965fe08-dc51-4875-8e04-08083ed934cf","html_url":"https://github.com/THUDM/P-tuning-v2","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUDM%2FP-tuning-v2","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUDM%2FP-tuning-v2/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUDM%2FP-tuning-v2/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUDM%2FP-tuning-v2/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/THUDM","download_url":"https://codeload.github.com/THUDM/P-tuning-v2/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243584532,"owners_count":20314782,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["natural-language-processing","p-tuning","parameter-efficient-learning","pretrained-language-model","prompt-tuning"],"created_at":"2024-07-30T20:00:56.199Z","updated_at":"2025-03-14T13:32:13.704Z","avatar_url":"https://github.com/THUDM.png","language":"Python","funding_links":[],"categories":["Models and Tools","Foundation Model Fine Tuning","预训练模型","Training","Papers","Python"],"sub_categories":["LLM Finetuning","Foundation Model Fine Tuning","Language Models"],"readme":"# P-tuning v2\n\n\nSource codes and data for\n* [ACL 2022] [P-Tuning v2: Prompt Tuning Can Be Comparable to Finetuning Universally Across Scales and Tasks](https://arxiv.org/abs/2110.07602) \n* [Findings of EMNLP 2023] [Parameter-Efficient Prompt Tuning Makes Generalized and Calibrated Neural Text Retrievers](https://arxiv.org/pdf/2207.07087.pdf)  [[Code]](https://github.com/THUDM/P-tuning-v2/tree/main/PT-Retrieval)\n\nAn optimized prompt tuning strategy achieving comparable performance to fine-tuning on small/medium-sized models and sequence tagging challenges. \n\nFind our previous version [P-tuning v1](https://github.com/THUDM/P-tuning) for knowledge probing and few-shot SuperGLUE. Your kindly starring our repo can greatly encourage us to work harder :)\n\nYou may be also interested in our recent work [GLM-130B: An Open Bilingual Pre-trained Model (2022-10-06)](https://arxiv.org/abs/2210.02414). It is an open-sourced LLM outperforming GPT-3 175B over various benchmarks. Get model weights, do inference and P-Tuning v2 with only **4 * RTX 3090 or 8 * RTX 2080 Ti** [FOR FREE](https://github.com/THUDM/GLM-130B)!\n\nP-tuning v2 leverages **deep prompt tuning**, which is to apply continuous prompts for every layer input of the pretrained transformer. \nDeep prompt tuning increases the capacity of continuous prompts and closes the gap to fine-tuning across various settings, especially for small models and hard tasks.\n\n![](figures/P-tuning-v2.png)\n\nThanks [@rainatam](https://github.com/rainatam)'s joint effort in re-organizing codes for publishing!\n\n## Commonly Asked Question\n1. Some readers notice a **'mismatch'** in SuperGLUE between P-tuning (v1) and P-tuning v2: This is because in P-tuning's SuperGLUE experiment, for fair comparison to PET, we follow its experimental setting where backbone pre-trained model parameters are jointly tuned with continuous prompt embeddings; while in P-tuning v2, we follow Prefix tuning and Lester et al.'s parameter-efficient setting where backbone pre-trained model parameters are frozen.\n\n## Reproduce Tips\nSince experiments reported in our paper are all conducted on NVIDIA DGX-A100 servers (which might be difficult to acquire), \nwe reimplement P-tuning v2's results on BERT-large/RoBERTa-large with:\n\n* Ubuntu servers with NVIDIA GeForce RTX 3090 (24G) GPUs\n* cuda 11.1\n* packages with certain versions (provided below)\n\nWe notice that the best hyper-parameters can be sensitive to your server environment and package version. \nIf you do not have the exact same environment, we highly recommend you to run hyper-parameter search in your environment\nbased on our example hyper-parameter search script in [search_script](search_script) and result collection scripts [search.py](search.py).\n\n### Setup\nWe conduct our experiment with Anaconda3. If you have installed Anaconda3, then create the environment for P-tuning v2:\n\n```shell\nconda create -n pt2 python=3.8.5\nconda activate pt2\n```\n\nAfter we setup basic conda environment, install pytorch related packages via:\n\n```shell\nconda install -n pt2 pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=11.0 -c pytorch\n```\n\nFinally, install other python packages we need:\n\n```shell\npip install -r requirements.txt\n```\n\n### Data\nFor SuperGLUE and SQuAD datasets, we download them from the Huggingface Datasets APIs (embedded in our codes).\n\nFor sequence tagging (NER, SRL) datasets, we prepare a non-official packup [here](https://zenodo.org/record/6318701/files/P-tuning-v2_data.tar.gz?download=1). \nAfter downloading, unzip the packup to the project root.\nPlease use at your own risk.\n\n### Training\nRun training scripts in [run_script](run_script) (e.g., RoBERTa for RTE):\n\n```shell\nbash run_script/run_rte_roberta.sh\n```\n\n### Implemented Results\nCurrently we have released our reimplementation on following tasks and datasets. More implementation will be released soon.\n\nReleased results on BERT-large\n\n|              | BoolQ | COPA | RTE  | WiC  | WSC  | CoNLL04 | OntoNotes 5.0 | CoNLL12 |\n|--------------|-------|------|------|------|------|---------|---------------|---------|\n| Result       | 74.3  | 77.0 | 80.1 | 75.1 | 68.3 | 84.5    | 86.4          | 85.3    |\n| Total Epochs | 100   | 80   | 60   | 80   | 80   | 40      | 30            | 45      |\n| Best Epoch   | 58    | 12   | 30   | 56   | 17   | 33      | 24            | 43      |\n\nReleased results on RoBERTa-large\n\n|              | BoolQ | COPA | RTE  | WiC  | WSC  | CoNLL03 | CoNLL04 | OntoNotes 5.0 | CoNLL12 | CoNLL05 WSJ | CoNLL05 Brown | SQuAD 1.1 | SQuAD 2.0 |\n|--------------|-------|------|------|------|------|---------|---------|---------------|---------|-------------|---------------|-----------|-----------|\n| Results      | 84.0  | 92.0 | 86.6 | 73.7 | 64.4 | 91.8    | 88.4    | 90.1          | 84.7    | 89.4        | 83.9          | 88.1/94.2 | 81.3/84.7 |\n| Total Epochs | 100   | 120  | 100  | 50   | 10   | 30      | 80      | 60            | 45      | 15          | -             | 30        | 10        |\n| Best Epoch   | 86    | 78   | 65   | 31   | 3    | 28      | 45      | 59            | 37      | 13          | -             | 24        | 9         |\n\nFor other hyper-parameters, please refer to the training scripts. \nIf you can not achieve the reported results at the best epoch, there is probably an environmental mismatch and hyper-parameter search is needed.\n\n## Citation\n\nIf you find our work useful, please kindly cite our paper:\n\n```\n@article{DBLP:journals/corr/abs-2110-07602,\n  author    = {Xiao Liu and\n               Kaixuan Ji and\n               Yicheng Fu and\n               Zhengxiao Du and\n               Zhilin Yang and\n               Jie Tang},\n  title     = {P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally\n               Across Scales and Tasks},\n  journal   = {CoRR},\n  volume    = {abs/2110.07602},\n  year      = {2021},\n  url       = {https://arxiv.org/abs/2110.07602},\n  eprinttype = {arXiv},\n  eprint    = {2110.07602},\n  timestamp = {Fri, 22 Oct 2021 13:33:09 +0200},\n  biburl    = {https://dblp.org/rec/journals/corr/abs-2110-07602.bib},\n  bibsource = {dblp computer science bibliography, https://dblp.org}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FTHUDM%2FP-tuning-v2","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FTHUDM%2FP-tuning-v2","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FTHUDM%2FP-tuning-v2/lists"}