{"id":27892618,"url":"https://github.com/dtyxs/pre-dpo","last_synced_at":"2026-06-23T21:31:46.721Z","repository":{"id":289516769,"uuid":"968592030","full_name":"DtYXs/Pre-DPO","owner":"DtYXs","description":"Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model","archived":false,"fork":false,"pushed_at":"2025-04-23T16:59:11.000Z","size":209,"stargazers_count":7,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-30T09:43:25.613Z","etag":null,"topics":["alignment","large-language-models","preference-optimization"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DtYXs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-18T11:10:22.000Z","updated_at":"2025-05-27T10:19:34.000Z","dependencies_parsed_at":"2025-04-23T18:00:04.667Z","dependency_job_id":null,"html_url":"https://github.com/DtYXs/Pre-DPO","commit_stats":null,"previous_names":["dtyxs/pre-dpo"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/DtYXs/Pre-DPO","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DtYXs%2FPre-DPO","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DtYXs%2FPre-DPO/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DtYXs%2FPre-DPO/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DtYXs%2FPre-DPO/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DtYXs","download_url":"https://codeload.github.com/DtYXs/Pre-DPO/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DtYXs%2FPre-DPO/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34708272,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-23T02:00:07.161Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["alignment","large-language-models","preference-optimization"],"created_at":"2025-05-05T13:28:57.835Z","updated_at":"2026-06-23T21:31:46.701Z","avatar_url":"https://github.com/DtYXs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Pre-DPO\n\n## Overview\n\nThis is the repository for the paper [Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model](https://arxiv.org/abs/2504.15843).\n\nPre-DPO is a simple yet effective DPO-based training paradigm that enhances preference optimization performance by leveraging a **guiding reference model**.\n\nThis repository is based on the popular repository [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory), which can easily fine-tune 100+ large language models.\n\n\n## Installation\n\nFirst, create a new conda environment and activate it.\n\n```shell\nconda create -n predpo python=3.10 \u0026\u0026 conda activate predpo\n```\n\nNext, clone the repository and install PyTorch along with the remaining dependencies.\n\n```shell\ngit clone https://github.com/DtYXs/Pre-DPO.git\ncd Pre-DPO\npip install torch==2.4.0 --index-url https://download.pytorch.org/whl/cu124\npip install -e \".[torch,metrics]\"\npip install deepspeed==0.15.4\n```\n\n## Training\n\n### Training Data\n\nFor the Base models ([Llama3.2-3B-Base](https://huggingface.co/meta-llama/Llama-3.2-3B) and [Qwen2.5-7B-Base](https://huggingface.co/Qwen/Qwen2.5-7B)), we utilize the [UltraChat-200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k) dataset to obtain the SFT models. Subsequently, we perform preference optimization using the [UltraFeedback-Binarized](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized) dataset.\n\nFor the Instruct models ([Llama3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct) and [Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct)), we follow the [pipeline](https://github.com/princeton-nlp/SimPO/tree/main/on_policy_data_gen) described in SimPO to generate on-policy preference data, using [ArmoRM-Llama3-8B-v0.1](https://huggingface.co/RLHFlow/ArmoRM-Llama3-8B-v0.1) as the preference label annotator. The resulting preference datasets are [llama3.2-3b-ultrafeedback-armorm-binarized](https://huggingface.co/datasets/DtYXs/llama3.2-3b-ultrafeedback-armorm-binarized) and [qwen2.5-7b-ultrafeedback-armorm-binarized](https://huggingface.co/datasets/DtYXs/qwen2.5-7b-ultrafeedback-armorm-binarized), respectively.\n\nYou can refer `./data/README.md` and prepare your data in `./data/dataset_info.json`.\n\n### Training Scripts\n\nWe provide our training scripts and examples in `./scripts`. We train the 3B models on 4 × 80G GPUs and the 7B models on 8 × 80G GPUs.\n\n#### SFT\n```shell\nbash scripts/train_sft.sh --model_name_or_path \u003cMODEL_NAME_OR_PATH\u003e --dataset \u003cDATASET_NAME\u003e --output_dir \u003cOUTPUT_DIR\u003e --template \u003cTEMPLATE\u003e\n```\n\n#### DPO\n```shell\nbash scripts/train_dpo.sh --sft_model_path \u003cSFT_MODEL_PATH\u003e --dataset \u003cDATASET_NAME\u003e --output_dir \u003cOUTPUT_DIR\u003e --template \u003cTEMPLATE\u003e --pref_beta \u003cBETA_IN_DPO\u003e --bsz \u003cBATCH_SIZE\u003e --gradient_accumulation_steps \u003cGRADIENT_ACCUMULATION_STEPS\u003e --lr \u003cLEARNING_RATE\u003e\n```\n\n#### SimPO\n```shell\nbash scripts/train_simpo.sh --sft_model_path \u003cSFT_MODEL_PATH\u003e --dataset \u003cDATASET_NAME\u003e --output_dir \u003cOUTPUT_DIR\u003e --template \u003cTEMPLATE\u003e --pref_beta \u003cBETA_IN_SIMPO\u003e --simpo_gamma \u003cGAMMA_IN_SIMPO\u003e --bsz \u003cBATCH_SIZE\u003e --gradient_accumulation_steps \u003cGRADIENT_ACCUMULATION_STEPS\u003e --lr \u003cLEARNING_RATE\u003e\n```\n\n#### Pre-DPO\n```shell\nbash scripts/train_predpo.sh --sft_model_path \u003cSFT_MODEL_PATH\u003e --ref_model_path \u003cREF_MODEL_PATH\u003e --dataset \u003cDATASET_NAME\u003e --output_dir \u003cOUTPUT_DIR\u003e --template \u003cTEMPLATE\u003e --pref_beta \u003cBETA_IN_PREDPO\u003e --bsz \u003cBATCH_SIZE\u003e --gradient_accumulation_steps \u003cGRADIENT_ACCUMULATION_STEPS\u003e --lr \u003cLEARNING_RATE\u003e\n```\n\n# Evaluation\n\nWe conduct evaluations on AlpacaEval 2.0 and Arena-Hard v0.1 following their official repositories.\n\n+ [AlpacaEval repository](https://github.com/tatsu-lab/alpaca_eval)\n+ [Arena-Hard repository](https://github.com/lmarena/arena-hard-auto)\n\n# Acknowledgement\n\nWe deeply appreciate the outstanding open-source code of [LlamaFactory](https://github.com/hiyouga/LLaMA-Factory) and [SimPO](https://github.com/princeton-nlp/SimPO), which has greatly supported research efforts within the community.\n\n# Citiation\n\nif Pre-DPO is helpful to your work, please cite our paper:\n```bibtex\n@misc{pan2025predpoimprovingdatautilization,\n      title={Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model}, \n      author={Junshu Pan and Wei Shen and Shulin Huang and Qiji Zhou and Yue Zhang},\n      year={2025},\n      eprint={2504.15843},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL},\n      url={https://arxiv.org/abs/2504.15843}, \n}\n```\n\n# Contact\n\n+ Email: panjunshu@westlake.edu.cn\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdtyxs%2Fpre-dpo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdtyxs%2Fpre-dpo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdtyxs%2Fpre-dpo/lists"}