{"id":31173282,"url":"https://github.com/general-preference/general-preference-model","last_synced_at":"2025-09-19T12:47:52.744Z","repository":{"id":258519829,"uuid":"866434458","full_name":"general-preference/general-preference-model","owner":"general-preference","description":"Official implementation of ICML 2025 paper \"Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment\" (https://arxiv.org/abs/2410.02197)","archived":false,"fork":false,"pushed_at":"2025-08-15T09:25:28.000Z","size":391,"stargazers_count":25,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-08-15T11:24:12.466Z","etag":null,"topics":["alignment","large-language-models","preference-modeling","preference-optimization","rlhf"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2410.02197","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/general-preference.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-10-02T08:54:42.000Z","updated_at":"2025-08-15T09:25:31.000Z","dependencies_parsed_at":"2025-07-25T09:19:46.685Z","dependency_job_id":"6b7417b2-7871-4925-8b66-576732c3d529","html_url":"https://github.com/general-preference/general-preference-model","commit_stats":null,"previous_names":["general-preference/general-preference-model"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/general-preference/general-preference-model","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/general-preference%2Fgeneral-preference-model","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/general-preference%2Fgeneral-preference-model/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/general-preference%2Fgeneral-preference-model/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/general-preference%2Fgeneral-preference-model/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/general-preference","download_url":"https://codeload.github.com/general-preference/general-preference-model/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/general-preference%2Fgeneral-preference-model/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275941523,"owners_count":25556976,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-19T02:00:09.700Z","response_time":108,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["alignment","large-language-models","preference-modeling","preference-optimization","rlhf"],"created_at":"2025-09-19T12:47:50.987Z","updated_at":"2025-09-19T12:47:52.733Z","avatar_url":"https://github.com/general-preference.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# General Preference Model (GPM)\n\n[![arXiv](https://img.shields.io/badge/arXiv-2410.02197-b31b1b.svg)](https://arxiv.org/abs/2410.02197) \n[![Website](https://img.shields.io/badge/Project-Website-green)](https://general-preference.github.io/general-preference-model) \n[![Huggingface Paper](https://img.shields.io/badge/Huggingface-Papers-blue)](https://huggingface.co/papers/2410.02197) \n\n[ICML 2025] Beyond Bradley-Terry Models: A General Preference\nModel for Language Model Alignment, [https://arxiv.org/abs/2410.02197](https://arxiv.org/abs/2410.02197). As Huggingface Daily Papers: [https://huggingface.co/papers/2410.02197](https://huggingface.co/papers/2410.02197)\n\n![](GPM.png)\n\n\n## Introduction\n\nThis repository is designed for training and evaluating the General Preference embedding model (GPM). It includes the following:\n\n* Training code for both GPM and BT reward models.\n\n* Evaluation code adapted from [RewardBench](https://github.com/allenai/reward-bench) for evaluating GPM and BT RM.\n\n* Code for LLM Alignment using GPM + GPO/SPPO: `./LLM-Alignment`.\n\n## Key Components:\n\n`scripts/run_train_rm_general_preference_single.sh`: Run training for reward models on a single node. `scripts/run_train_rm_general_preference_multi.sh`: Run training for reward models across multiple nodes.  \n`rewardbench_eval/run_rm_rewardbench.sh`: Run RewardBench evaluations for reward models.\n`eval/batch_inference_rm_general_preference.sh`: Run evaluations on a custom-defined dataset.\n`general_preference`: Useful code for training, heavily based on [OpenRLHF](https://github.com/OpenRLHF/OpenRLHF).\n\n## Quick Start\n### Installation\nrewardbench==0.1.2 depends on transformers==4.44.0. Install rewardbench first to avoid dependency conflicts. \n```bash\ngit clone https://github.com/general-preference/general-preference-model\ncd general-preference-model\npip install rewardbench==0.1.2\npip install -e .\n```\nReinstall deepspeed with specific build options. \n```bash\nexport DS_BUILD_SPARSE_ATTN=0; export DS_BUILD_EVOFORMER_ATTN=0; DS_BUILD_OPS=1 pip install deepspeed==0.13.5 --no-cache --force-reinstall\n```\nIf the installation fails, install torch before other packages.\n```bash\npip install torch==2.3.0\npip install -e .\n```\n\n## Example Usage of the GPM\n\nBelow is an example code snippet (see `./gpm_example_usage.py`):\n\n```python\nprompt_text = \"Describe the importance of reading books in today's digital age.\"\nresponse1 = \"Books remain crucial in the digital era, offering in-depth knowledge and fostering critical thinking. They provide a unique, immersive experience that digital media can't replicate, contributing significantly to personal and intellectual growth.\"\nresponse2 = \"Books are still useful for learning new things. They help you relax and can be a good break from screens.\"\n\ncontext1 = [\n    {\"role\": \"user\", \"content\": prompt_text},\n    {\"role\": \"assistant\", \"content\": response1}\n]\n\ncontext2 = [\n    {\"role\": \"user\", \"content\": prompt_text},\n    {\"role\": \"assistant\", \"content\": response2}\n]\n\nrm = GPMPipeline(\"general-preference/GPM-Llama-3.1-8B\", value_head_dim=6)\n\nreward1, prompt_hidden_state = rm([context1], return_prompt=True)\nreward2 = rm([context2])\n\nresult = generate_high_dim_result_with_prompt(rm.model, rm.value_head_dim, reward1, reward2, prompt_hidden_state)\n\nresult_batch = result.float().cpu().detach().numpy().tolist()\n\nresults = []\n[\n    results.append(1) if result \u003e 0 else results.append(0)\n    for result in result_batch\n]\n\nprint(result_batch)\n```\n\n## Citations\n\nPlease cite the paper and star this repo if you use the General Preference embedding Model (GPM) and General Preference Optimization (GPO) and find it interesting/useful, thanks! Feel free to contact zhangge19951114@gmail.com | yifanzhangresearch@gmail.com or open an issue if you have any questions.\n\n```\n@inproceedings{zhang2024beyond,\n  title={Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment},\n  author={Zhang, Yifan and Zhang, Ge and Wu, Yue and Xu, Kangping and Gu, Quanquan},\n  booktitle={Forty-second International Conference on Machine Learning}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeneral-preference%2Fgeneral-preference-model","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgeneral-preference%2Fgeneral-preference-model","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeneral-preference%2Fgeneral-preference-model/lists"}