{"id":13564069,"url":"https://github.com/openai/lm-human-preferences","last_synced_at":"2025-04-08T16:07:22.881Z","repository":{"id":52303246,"uuid":"208397992","full_name":"openai/lm-human-preferences","owner":"openai","description":"Code for the paper Fine-Tuning Language Models from Human Preferences","archived":false,"fork":false,"pushed_at":"2023-07-25T16:48:57.000Z","size":97,"stargazers_count":1308,"open_issues_count":16,"forks_count":168,"subscribers_count":22,"default_branch":"master","last_synced_at":"2025-04-01T15:09:52.115Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://openai.com/blog/fine-tuning-gpt-2/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-09-14T06:15:53.000Z","updated_at":"2025-04-01T03:41:20.000Z","dependencies_parsed_at":"2024-01-14T03:51:00.712Z","dependency_job_id":"c66b5a6f-ec6d-4818-8c41-7d972321cb72","html_url":"https://github.com/openai/lm-human-preferences","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openai%2Flm-human-preferences","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openai%2Flm-human-preferences/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openai%2Flm-human-preferences/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openai%2Flm-human-preferences/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openai","download_url":"https://codeload.github.com/openai/lm-human-preferences/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247878022,"owners_count":21011158,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T13:01:26.240Z","updated_at":"2025-04-08T16:07:22.860Z","avatar_url":"https://github.com/openai.png","language":"Python","funding_links":[],"categories":["Python","📚 Resources \u0026 Tutorials","Papers","🛠️ Libraries \u0026 Implementations"],"sub_categories":["Recent Advances (2023–2025)","2020 and before"],"readme":"**Status:** Archive (code is provided as-is, no updates expected)\n\n**Status:** All references to `gs://lm-human-preferences/` were updated to `https://openaipublic.blob.core.windows.net/lm-human-preferences`, as we migrated from GCP to Azure. The code provided as is may no longer work. Pull requests welcome\n\n# lm-human-preferences\n\nThis repository contains code for the paper [Fine-Tuning Language Models from Human Preferences](https://arxiv.org/abs/1909.08593).  See also our [blog post](https://openai.com/blog/fine-tuning-gpt-2/).\n\nWe provide code for:\n- Training reward models from human labels\n- Fine-tuning language models using those reward models\n\nIt does not contain code for generating labels.  However, we have released human labels collected for our experiments, at `gs://lm-human-preferences/labels`.\nFor those interested, the question and label schemas are simple and documented in [`label_types.py`](./lm_human_preferences/label_types.py).\n\nThe code has only been tested using the smallest GPT-2 model (124M parameters).\n\n## Instructions\n\nThis code has only been tested using Python 3.7.3.  Training has been tested on GCE machines with 8 V100s, running Ubuntu 16.04, but development also works on Mac OS X.\n\n### Installation\n\n- Install [pipenv](https://github.com/pypa/pipenv#installation).\n\n- Install [tensorflow](https://www.tensorflow.org/install/gpu):  Install CUDA 10.0 and cuDNN 7.6.2, then `pipenv install tensorflow-gpu==1.13.1`.  The code may technically run with tensorflow on CPU but will be very slow.\n\n- Install [`gsutil`](https://cloud.google.com/storage/docs/gsutil_install)\n\n- Clone this repo.  Then:\n  ```\n  pipenv install\n  ```\n\n- (Recommended) Install [`horovod`](https://github.com/horovod/horovod#install) to speed up the code, or otherwise substitute some fast implementation in the `mpi_allreduce_sum` function of [`core.py`](./lm_human_preferences/utils/core.py).  Make sure to use pipenv for the install, e.g. `pipenv install horovod==0.18.1`.\n\n### Running\n\nThe following examples assume we are aiming to train a model to continue text in a physically descriptive way.\nYou can read [`launch.py`](./launch.py) to see how the `descriptiveness` experiments and others are defined.\n\nNote that we provide pre-trained models, so you can skip directly to RL fine-tuning or even to sampling from a trained policy, if desired.\n\n#### Training a reward model\n\nTo train a reward model, use a command such as\n```\nexperiment=descriptiveness\nreward_experiment_name=testdesc-$(date +%y%m%d%H%M)\npipenv run ./launch.py train_reward $experiment $reward_experiment_name\n```\n\nThis will save outputs (and tensorboard event files) to the directory `/tmp/save/train_reward/$reward_experiment_name`.  The directory can be changed via the `--save_dir` flag.\n\n#### Finetuning a language model\n\nOnce you have trained a reward model, you can finetune against it.\n\nFirst, set\n```\ntrained_reward_model=/tmp/save/train_reward/$reward_experiment_name\n```\nor if using our pretrained model,\n```\ntrained_reward_model=gs://lm-human-preferences/runs/descriptiveness/reward_model\n```\n\nThen,\n```\nexperiment=descriptiveness\npolicy_experiment_name=testdesc-$(date +%y%m%d%H%M)\npipenv run ./launch.py train_policy $experiment $policy_experiment_name --rewards.trained_model $trained_reward_model --rewards.train_new_model 'off'\n```\n\nThis will save outputs (and tensorboard event files) to the directory `/tmp/save/train_policy/$policy_experiment_name`.  The directory can be changed via the `--save_dir` flag.\n\n#### Both steps at once\n\nYou can run a single command to train a reward model and then finetune against it\n```\nexperiment=descriptiveness\nexperiment_name=testdesc-$(date +%y%m%d%H%M)\npipenv run ./launch.py train_policy $experiment $experiment_name\n```\n\nIn this case, outputs are in the directory `/tmp/save/train_policy/$policy_experiment_name`, and the reward model is saved to a subdirectory `reward_model`.  The directory can be changed via the `--save_dir` flag.\n\n#### Sampling from a trained policy\n\nSpecify the policy to load:\n```\nsave_dir=/tmp/save/train_policy/$policy_experiment_name\n```\nor if using our pretrained model,\n```\nsave_dir=gs://lm-human-preferences/runs/descriptiveness\n```\n\nThen run:\n```\npipenv run ./sample.py sample --save_dir $save_dir --savescope policy\n```\n\nNote that this script can run on less than 8 GPUs.  You can pass the flag `--mpi 1`, for exapmle, if you only have one GPU.\n\n## LICENSE\n\n[MIT](./LICENSE)\n\n## Citation\n\nPlease cite the paper with the following bibtex entry:\n```\n@article{ziegler2019finetuning,\n  title={Fine-Tuning Language Models from Human Preferences},\n  author={Ziegler, Daniel M. and Stiennon, Nisan and Wu, Jeffrey and Brown, Tom B. and Radford, Alec and Amodei, Dario and Christiano, Paul and Irving, Geoffrey},\n  journal={arXiv preprint arXiv:1909.08593},\n  url={https://arxiv.org/abs/1909.08593},\n  year={2019}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenai%2Flm-human-preferences","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenai%2Flm-human-preferences","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenai%2Flm-human-preferences/lists"}