{"id":31211517,"url":"https://github.com/cyberagentailab/filtered-dpo","last_synced_at":"2025-09-21T05:30:24.376Z","repository":{"id":235039384,"uuid":"786694117","full_name":"CyberAgentAILab/filtered-dpo","owner":"CyberAgentAILab","description":"Introducing Filtered Direct Preference Optimization (fDPO) that enhances language model alignment with human preferences by discarding lower-quality samples compared to those generated by the learning model","archived":false,"fork":false,"pushed_at":"2024-11-27T09:49:40.000Z","size":252,"stargazers_count":16,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-10T07:42:49.593Z","etag":null,"topics":["alignment","dpo","rlhf"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2404.13846","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CyberAgentAILab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-04-15T06:03:47.000Z","updated_at":"2025-08-27T04:07:23.000Z","dependencies_parsed_at":"2025-09-10T05:38:55.131Z","dependency_job_id":"a7f7334c-acbd-46f5-a01a-359486efd28b","html_url":"https://github.com/CyberAgentAILab/filtered-dpo","commit_stats":null,"previous_names":["cyberagentailab/filtered-dpo"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/CyberAgentAILab/filtered-dpo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyberAgentAILab%2Ffiltered-dpo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyberAgentAILab%2Ffiltered-dpo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyberAgentAILab%2Ffiltered-dpo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyberAgentAILab%2Ffiltered-dpo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CyberAgentAILab","download_url":"https://codeload.github.com/CyberAgentAILab/filtered-dpo/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyberAgentAILab%2Ffiltered-dpo/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":276195627,"owners_count":25601152,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-21T02:00:07.055Z","response_time":72,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["alignment","dpo","rlhf"],"created_at":"2025-09-21T05:30:22.788Z","updated_at":"2025-09-21T05:30:24.370Z","avatar_url":"https://github.com/CyberAgentAILab.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Filtered Direct Preference Optimization\n\n### tl;dr\n Introducing Filtered Direct Preference Optimization (fDPO) that enhances language model alignment with human preferences by discarding lower-quality samples compared to those generated by the learning model\n\n## Prerequisites\n- [Python 3.10.x][python]  \n- [Poetry 1.7.x][poetry]  \n- [direnv][direnv] \n\n[python]: https://www.python.org/downloads/release/python-31012/\n[poetry]: https://python-poetry.org/\n[direnv]: https://direnv.net/\n\n## Get Started\n\nTo set up your local environment, start by copying the example environment file:\n\n```shell\ncp .env.example .env\n```\n\nNext, you need to edit the .env file to include your Hugging Face API token. Replace the placeholder value with your actual token:\n\n```\nHF_HUB_TOKEN=\"your_hugging_face_token_here\"\n```\n\nIf you do not already have a Hugging Face account or API token, you will need to create an account on Hugging Face and then generate an API token from your account settings.\n\nOnce your .env file is set up, apply the configuration to your environment using direnv:\n\n```shell\ndirenv allow .\n```\n\n### Installation\n```shell\npoetry install\n```\n\n### Obtain Access to Datasets and Models\n\nTo use the datasets and models listed below, you must apply for access privileges on their respective Hugging Face repository pages. Please follow the links provided, and on each page, click the “Apply” button to submit your access request. This process is necessary to ensure compliance with the data usage policies and intellectual property rights associated with each resource.\n\n\n- [Dataset][dataset] - Follow this link to apply for access to the dataset.\n- [Model][model] - Follow this link to apply for access to the model.\n\n\n[dataset]: https://huggingface.co/datasets/Mitsuki-Sakamoto/fdpo-preference-dataset\n[model]: https://huggingface.co/Mitsuki-Sakamoto/fdpo-models\n\n## Usage\n\n### Test training\nExecution time of about an hour in the notebook.\n```\nbash scripts/test.sh \n```\n\n### Train 160m model\nExecution time of several hours using A100 80G\n```\n# $seed in {1, 2, 3}\nseed=1\nbash scripts/160m/fdpo_mix.sh ${seed}\n```\n\n\n### Train 1.4b model\nExecution time of about a day using A100 80G\n```\n# $seed in {1, 2, 3}\nseed=1\nbash scripts/1.4b/fdpo_mix.sh ${seed}\n```\n\n\n## Checking Experimental Results\nThe verification of experiment logs and creation of reports follow the standard of [Transformers](https://huggingface.co/docs/transformers/v4.39.3/en/main_classes/callback#callbacks) .\n\nAlso, a notebook for reproducing Figure 6 in our paper is provided in [notebook](notebook/fDPO-propostion.ipynb)\n\n## Reference\n\n[Morimura, T., Sakamoto, M., Jinnai, Y., Abe, K., and Ariu, K., Filtered Direct Preference Optimization. EMNLP, 2024.](https://arxiv.org/abs/2404.13846)\n\nBibtex:\n```\n@inproceedings{morimura-etal-2024-filtered,\n    title = \"Filtered Direct Preference Optimization\",\n    author = \"Morimura, Tetsuro  and\n      Sakamoto, Mitsuki  and\n      Jinnai, Yuu  and\n      Abe, Kenshi  and\n      Ariu, Kaito\",\n    editor = \"Al-Onaizan, Yaser  and\n      Bansal, Mohit  and\n      Chen, Yun-Nung\",\n    booktitle = \"Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing\",\n    month = nov,\n    year = \"2024\",\n    address = \"Miami, Florida, USA\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2024.emnlp-main.1266\",\n    pages = \"22729--22770\",\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcyberagentailab%2Ffiltered-dpo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcyberagentailab%2Ffiltered-dpo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcyberagentailab%2Ffiltered-dpo/lists"}