{"id":18887148,"url":"https://github.com/thunlp-mt/pgra","last_synced_at":"2025-09-04T08:32:51.769Z","repository":{"id":192023347,"uuid":"640899132","full_name":"THUNLP-MT/PGRA","owner":"THUNLP-MT","description":"Prompt-Guided Retrieval For Non-Knowledge-Intensive Tasks","archived":false,"fork":false,"pushed_at":"2023-09-01T09:10:53.000Z","size":34,"stargazers_count":9,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-11-08T07:40:53.789Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/THUNLP-MT.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-05-15T11:08:50.000Z","updated_at":"2024-07-11T02:38:25.000Z","dependencies_parsed_at":"2023-09-02T05:35:46.221Z","dependency_job_id":"c9218936-039d-4b2f-9ead-7439ee961480","html_url":"https://github.com/THUNLP-MT/PGRA","commit_stats":null,"previous_names":["thunlp-mt/pgra"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FPGRA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FPGRA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FPGRA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FPGRA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/THUNLP-MT","download_url":"https://codeload.github.com/THUNLP-MT/PGRA/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":231945471,"owners_count":18449892,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T07:34:46.666Z","updated_at":"2024-12-31T05:18:59.913Z","avatar_url":"https://github.com/THUNLP-MT.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PGRA\nPrompt-Guided Retrieval For Non-Knowledge-Intensive Tasks\n\nThis is the code for [Prompt-Guided Retrieval For Non-Knowledge-Intensive Tasks](https://arxiv.org/abs/2305.17653)\n\n## Data\nIn this work, we use the Wiki1m dataset in SimCSE as the retrieval dataset. You can download this dataset from the [downloading scripts](https://github.com/princeton-nlp/SimCSE/blob/main/data/download_wiki.sh) provided by them.\n\nAs for the training and testing data, we follow the setup of [LM-BFF](https://github.com/princeton-nlp/SimCSE/blob/main/data/download_wiki.sh). Please follow the data preparation process provided by them. After processing the data, you will get a folder named k-shot with training and testing files of all tasks inside. \n\n## First Stage Retrieval\nWe provide first-stage retrieval with BM25, BERT and SimCSE. The codes for each retriever are in `get_data_bm25`, `get_data_bert`, `get_data_simcse` respectively. \n\nFirstly, embeddings of sentences in Wiki1m need to be generated if you use BERT or SimCSE. For example, if you want to use the BERT model as the retriever. You can run the following code:\n```\nWIKI1M_PATH=\nOUTPUT_FILE=\nMODEL_NAME_OR_PATH=bert-base-uncased\n\npython get_data_bert/sbert.py \\\n    --wiki1m_path $WIKI1M_PATH \\\n    --output_sents_file $OUTPUT_FILE \\\n    --sbert $MODEL_NAME_OR_PATH\n```\nAn embedding file of Wiki1m with BERT will be save to `OUTPUT_FILE`.\n\nThen, this embedding file can be used to perform retrieval. To do this, you need to firstly fill in the train and test files of tasks in the `single_sentence.py` file in each folder. Then, take BERT retriever as an example, you can run the following command:\n```\ntask=\npython get_data_bert/single_sentence.py ${task}\n```\nThis will generate the top 150 retrieved evidence for each sentence in the task and save this into `${task}.train.json` or `${task}.test.json`.\n\nWe add examples of final CoLA examples in the `get_data_bert` folder.\n\n\n## Second-stage retrieval\nIn this stage, pretrained language models (in our case OPT models) are used to further rerank the retrieval process. For efficiency purposes, we split this process into two stages. \n\nOn the first stage, the OPT models are used to compute embeddings of retrieved evidence for a given task. Due to the fact that many retrieved evidence will appear more than once in a task, this step can save a lot of time. To run this, run the following command:\n```\nexport CUDA_VISIBLE_DEVICES=\ntask=\nSPLIT=\nMODEL_NAME_OR_PATH=\n\npython second_stage.py \\\n        --task $task \\\n        --split $SPLIT \\\n        --model_name_or_path $MODEL_NAME_OR_PATH \\\n        --src_tag bert \\\n        --save_tag bert \\\n        --d -1\n```\nThis script will process `get_data_bert/${task}.${SPLIT}.json` file and output `get_data_2stage_fewshot/bert_${MODEL_NAME}/${TASK}-${SPLIT}_ctxs_embeddings.pt`.\n\nAfter this, this embedding can be used to generate the rerank the output. To do this, run\n```\npython write_from_embeddings.py \\\n    --task $task \\\n    --split $split \\\n    --model opt-13b \\\n    --src_tag bert \\\n    --embedding_tag bert \\\n    --save_tag bert\n    --d -1\n```\n\nAt this time, the model is the stem of `MODEL_NAME_OR_PATH` at the last step. This time, files generated before will be used and final reranked evidence will be saved at `get_data_2stage_fewshot/bert_${MODEL_NAME}/${TASK}.${SPLIT}.json`. We also provided such example files in the `get_data_2stage_fewshot/bert_opt-13b` folder. \n\n\n## Train the reader\nWe copy the [FID](https://github.com/facebookresearch/FiD) implementation to train the reader. Before doing this, you may need to split the train file into train and dev splits.\n\nTo run the trainer, you can run the following command:\n```\nSTEPS=5000\nREADER=2stage_fewshot/bert_opt-13b\nexport TRANSFORMERS_VERBOSITY=error\n\nLR=1e-4\nBSZ=8\nK=16\nTASKS=cola\nSEED=\nT5_path=\n\npython  train_reader.py \\\n    --tasks ${TASKS} \\\n    --name ${READER}/${TASKS} \\\n    --train_data get_data_${READER}/${TASKS}.train.json \\\n    --eval_data get_data_${READER}/${TASKS}.dev.json \\\n    --model_name_or_path $T5_path \\\n    --model_size base \\\n    --seed $seed \\\n    --lr ${LR} \\\n    --per_gpu_batch_size ${BSZ} \\\n    --total_step ${STEPS} \\\n    --n_context ${K} \\\n    --optim adamw \\\n    --scheduler linear \\\n    --weight_decay 0.01 \\\n    --text_maxlength 128 \\\n    --warmup_step 0\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthunlp-mt%2Fpgra","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthunlp-mt%2Fpgra","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthunlp-mt%2Fpgra/lists"}