{"id":13678331,"url":"https://github.com/Albert-Ma/PROP","last_synced_at":"2025-04-29T13:30:25.834Z","repository":{"id":37640093,"uuid":"304513239","full_name":"Albert-Ma/PROP","owner":"Albert-Ma","description":"WSDM'2021, PROP and SIGIR'2021,B-PROP","archived":false,"fork":false,"pushed_at":"2023-05-18T15:41:32.000Z","size":1383,"stargazers_count":110,"open_issues_count":4,"forks_count":18,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-11-10T14:51:19.366Z","etag":null,"topics":["b-prop","pretrain-for-search","pretraining-for-ir","prop","representative-words-prediction"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Albert-Ma.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-10-16T03:49:48.000Z","updated_at":"2024-04-14T07:20:59.000Z","dependencies_parsed_at":"2023-01-25T10:32:37.223Z","dependency_job_id":"aafe0158-ef0a-49c4-949a-f0f8d3abf84a","html_url":"https://github.com/Albert-Ma/PROP","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Albert-Ma%2FPROP","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Albert-Ma%2FPROP/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Albert-Ma%2FPROP/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Albert-Ma%2FPROP/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Albert-Ma","download_url":"https://codeload.github.com/Albert-Ma/PROP/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224173772,"owners_count":17268174,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["b-prop","pretrain-for-search","pretraining-for-ir","prop","representative-words-prediction"],"created_at":"2024-08-02T13:00:52.440Z","updated_at":"2024-11-11T20:31:25.039Z","avatar_url":"https://github.com/Albert-Ma.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# PROP\n\n\n## Introduction\n**PROP**, **P**re-training with **R**epresentative w**O**rds **P**rediction, is a new pre-training method tailored for ad-hoc retrieval. PROP is inspired by the classical statistical language model for IR, specifically the query likelihood model, which assumes that the query is generated as the piece of text representative of the “ideal” document. Based on this idea, we construct the representative words prediction (ROP) task for pre-training. The full paper can be found [here](https://arxiv.org/pdf/2010.10137.pdf).\n\n---\n\n* 🔥 2022-7 Update: we have uploaded the PROP model to the huggingface model cards 🤗.\n* 🔥 2021-8 Update: we add the [B-PROP](bprop/README.md) pre-training in this repo.\n* 🔥 **News 2021-1-7: PROP_step400k base (ensemble v0.1) got first place on [MS MARCO Document Ranking Leaderboard](https://microsoft.github.io/MSMARCO-Document-Ranking-Submissions/leaderboard/).**\n\n\n---\n\n\n## Pre-trained models in the Huggingface 🤗\n\nWe have uploaded three PROP models to the Huggingface Hub, so you can easily use the PROP models with [Huggingface/Transformers](https://github.com/huggingface/transformers) library.\n\nModel identifier in Huggingface Hub:\n\n- `xyma/PROP-wiki`: The official PROP model pre-trained on Wikipedia and used in our WSDM'2021 paper\n- `xyma/PROP-marco`: The official PROP model pre-trained on MS MARCO document corpus and used in our WSDM'2021 paper\n- `xyma/PROP-marco-step400k`: PROP model pre-trained on MS MARCO document corpus with more steps and used in MS MARCO Document Ranking Leaderboard\n\nFor example,\n```\ntokenizer = AutoTokenizer.from_pretrained(\"xyma/PROP-marco\")\nmodel = AutoModel.from_pretrained(\"xyma/PROP-marco\")\n```\n\n\u003e Note that if you want to test the zero-shot performance of PROP models, please use this repo or create a PROP model like [this](https://github.com/Albert-Ma/PROP/blob/main/pytorch_pretrain_bert/modeling.py#L775) since we use a different CLS head with BERT.\nThe CLS head identifier in PROP model weights is only `cls` which is a linear module (768, 1).\n\nFor example, you can ceate PROP class like the following before fine-tuning,\n```\nfrom transformers import BertPreTrainedModel, BertModel, BertLMPredictionHead\n\nclass PROP(BertPreTrainedModel):\n  def __init__(self, config):\n      super(PROP, self).__init__(config)\n      self.bert = BertModel(config)\n      self.cls = torch.nn.Linear(config.hidden_size, 1)\n      self.cls.predictions = BertLMPredictionHead(config, self.bert.embeddings.word_embeddings.weight)\n      self.init_weights()\n  ...\n```\n\nAlso you can download the PROP models from the Google drive:\n\n- `PROP-wiki`: [download link](https://drive.google.com/file/d/11uj30VgEsVLj6PekP-SBvjWzlyLDP7Hf/view?usp=sharing)\n- `PROP-marco`: [download link](https://drive.google.com/file/d/1E2E-kS_gXp28imhjNdNGNX8EFGt5cs-5/view?usp=sharing)\n- `PROP-marco-step400k`: [download link](https://drive.google.com/file/d/1aw0s1UK8PvZCI9R8hA9b7kxoN0x35kRr/view?usp=sharing)\n\n\n\n## Pre-training Corpus\n\n### Download data\nFor **Wikipedia**, download [the dump](https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2) and extract the text with [`WikiExtractor.py`](https://github.com/attardi/wikiextractor), and then apply any necessary cleanup (e.g. remove spaces and special characters) to convert it into clean text.\nFor **MS MARCO**, download corpus from the official TREC [website](https://msmarco.blob.core.windows.net/msmarcoranking/msmarco-docs.tsv.gz).\n\n### Pre-process data\nCompute and store the statistics about `tf, df, average_word_num et.al` on the whole corpus into `Json` file. Also, you can use the IR toolkit like lucene to get these info.\n\nRun `./scripts/preprocess.sh` to generate the following files:\n\n* `corpus_df_file.json`: `{word: document tf}`\n* `doc_tf_file.json`: `{doc_id, doc_tf, doc_word_num}`, one document per line\n* `corpus_tf_file.json`: `{word: corpus tf}`\n* `info_file.json`: `{total_doc_num, total_word_num, average_doc_word_num}`\n* `stem2pos_file.json`: `{stem: {word: count}}`\n* `preprocessed_data`: `{docid, bert_tokenized_doc_text}`\n\n### Generate representive word sets\n\nSee `./scripts/generate_word_sets.sh` \n\nThe details are as follows and you can tune the hyperparameter `possion_lambda` to match with your target dataset (the average of query length):\n\n```shell\nexport INPUT_FILE=/path/to/preprocessed_data\nexport Bert_MODEL_DIR=/path/to/pytorch_version/bert-base/\nexport OUTPUT=/path/to/output\n\npython multiprocessing_generate_word_sets.py \\\n    --train_corpus $INPUT_FILE  \\\n    --do_lower_case \\\n    --bert_model $Bert_MODEL_DIR \\\n    --output_dir $OUTPUT \\\n    --epochs_to_generate 1 \\\n    --possion_lambda 3 \\\n    --rop_num_per_doc 10 \\\n    --num_workers 20 \\\n    --reduce_memory\n```\n\n### Generate training instances\n\nSee `./scripts/generate_pair_instances.sh` \n\nThe details are as follows and other hyperparameters are set to the default value:\n\n```shell\nexport INPUT_FILE=/path/to/preprocessed_data\nexport Bert_MODEL_DIR=/path/to/pytorch_version/bert-base/\nexport OUTPUT=/path/to/output\n\npython multiprocessing_generate_pairwise_instances.py \\\n    --train_corpus $INPUT_FILE \\\n    --bert_model $Bert_MODEL_DIR \\\n    --do_lower_case \\\n    --output_dir $OUTPUT \\\n    --epochs_to_generate 1 \\\n    --rop_num_per_doc 4 \\\n    --mlm \\\n    --max_seq_len 512 \\\n    --num_workers 20 \\\n    --reduce_memory\n```\n\n## Pre-training\n\nSee `./scripts/run_pretrain.sh` \n\n```shell\nexport DATA_DIR=/path/to/pretraining_data\nexport Bert_MODEL_DIR=/path/to/pytorch_version/bert-base/\nexport OUTPUT=/path/to/output\n\nCUDA_VISIBLE_DEVICES=\"0,1,2,3\" \\\npython run_pretraining.py \\\n        --pregenerated_data $DATA_DIR \\\n        --bert_model $Bert_MODEL_DIR \\\n        --do_lower_case \\\n        --output_dir $OUTPUT \\\n        --train_batch_size 80\\\n        --save_checkpoints_steps 1000\\\n        --gradient_accumulation_steps 2-5\\\n        --epochs 1 \\\n        --negtive_num 1 \\\n        --learning_rate 2e-5\\\n        --reduce_memory\n```\n\n\n\n## Fine-tuning\nPROP have the same architecture with [`BERT-Base`](https://github.com/google-research/bert), and thus you can fine-tune PROP like BERT on any downstream ad-hoc retrieval tasks by just replacing BERT checkpoints with PROP's.\n\n```\nclass ReRanker(torch.nn.Module):\n    def __init__(self, model_name_or_path):\n        super().__init__()\n\n        self.config = AutoConfig.from_pretrained(model_name_or_path, num_labels=1)\n        self.prop = AutoModelForSequenceClassification.from_pretrained(model_name_or_path, config=self.config)\n\n    def forward(self, input_ids, token_type_ids, input_mask, labels=None, group_num=2):\n        batch_size = input_ids.size(0)\n        output  = self.prop(input_ids, attention_mask=input_mask, token_type_ids=token_type_ids, return_dict=True)\n        logits = output['logits']\n\n        if labels is not None:\n            logits = logits.reshape(batch_size//group_num, group_num)\n            pairwise_labels = torch.zeros(batch_size//group_num, dtype=torch.long).to(logits.device)\n            pairwise_loss = self.cross_entropy(logits, pairwise_labels)\n            return pairwise_loss\n        else:\n            return logits\n```\n\n\n## Citation\nIf you find our work useful, please consider citing our paper:\n```\n@article{ma2020prop,\n  title={PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrieval},\n  author={Ma, Xinyu and Guo, Jiafeng and Zhang, Ruqing and Fan, Yixing and Ji, Xiang and Cheng, Xueqi},\n  journal={Proceedings of the 14th ACM International Conference on Web Search and Data Mining},\n  year={2021}\n}\n\n@article{ma2020bprop,\n  title={B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrieval},\n  author={Ma, Xinyu and Guo, Jiafeng and Zhang, Ruqing and Fan, Yixing and Ji, Xiang and Cheng, Xueqi},\n  journal={Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval},\n  year={2021}\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAlbert-Ma%2FPROP","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FAlbert-Ma%2FPROP","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAlbert-Ma%2FPROP/lists"}