{"id":13648100,"url":"https://github.com/tatsu-lab/alpaca_farm","last_synced_at":"2025-04-12T18:44:12.921Z","repository":{"id":168702782,"uuid":"635992742","full_name":"tatsu-lab/alpaca_farm","owner":"tatsu-lab","description":"A simulation framework for RLHF and alternatives. Develop your RLHF method without collecting human data. ","archived":false,"fork":false,"pushed_at":"2024-07-01T18:53:35.000Z","size":1895,"stargazers_count":804,"open_issues_count":6,"forks_count":62,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-04-03T20:12:05.284Z","etag":null,"topics":["deep-learning","instruction-following","large-language-models","natural-language-processing","reinforcement-learning-from-human-feedback"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2305.14387","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tatsu-lab.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-03T22:44:52.000Z","updated_at":"2025-04-03T14:30:40.000Z","dependencies_parsed_at":"2023-11-23T05:41:36.199Z","dependency_job_id":"33729494-779c-4c52-b8fc-e1c03033b1c9","html_url":"https://github.com/tatsu-lab/alpaca_farm","commit_stats":null,"previous_names":["tatsu-lab/alpaca_farm"],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tatsu-lab%2Falpaca_farm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tatsu-lab%2Falpaca_farm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tatsu-lab%2Falpaca_farm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tatsu-lab%2Falpaca_farm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tatsu-lab","download_url":"https://codeload.github.com/tatsu-lab/alpaca_farm/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248617199,"owners_count":21134190,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","instruction-following","large-language-models","natural-language-processing","reinforcement-learning-from-human-feedback"],"created_at":"2024-08-02T01:03:58.282Z","updated_at":"2025-04-12T18:44:12.888Z","avatar_url":"https://github.com/tatsu-lab.png","language":"Python","funding_links":[],"categories":["Python","A01_文本生成_文本对话","Industry Strength RL","Papers"],"sub_categories":["大语言对话模型及数据","2023"],"readme":"\u003cp align=\"center\" width=\"100%\"\u003e\n\u003cimg src=\"assets/AlpacaFarm_big.png\" alt=\"AlpacaFarm\" style=\"width: 50%; min-width: 300px; display: block; margin: auto;\"\u003e\n\u003c/p\u003e\n\n# AlpacaFarm: A Simulation Framework for Methods that \u003cbr/\u003eLearn from Human Feedback\n\n[![Code License](https://img.shields.io/badge/Code%20License-Apache_2.0-green.svg)](https://github.com/tatsu-lab/alpaca_farm/blob/main/LICENSE)\n[![Data License](https://img.shields.io/badge/Data%20License-CC%20By%20NC%204.0-red.svg)](https://github.com/tatsu-lab/alpaca_farm/blob/main/DATA_LICENSE)\n[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/release/python-3100/)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n\n\n**Changing auto-annotators**: `text-davinci-003` is [now depreciated](https://platform.openai.com/docs/deprecations) by OpenAI, as a result, we can't use the original pool of annotators for automatically generating preferences (for fine-tuning or evaluation). We, therefore, switched to the GPT-4 annotator from [AlpacaEval 1](https://github.com/tatsu-lab/alpaca_eval). All results should thus be compared to models from AlpacaEval 1 rather than the original AlpacaFarm results. Note that over-optimization might not be seen in this new setting (see Figure 4 in the [paper](https://arxiv.org/abs/2305.14387)). We are sorry for the inconvenience caused.\n\n---\n\nResearch and development on learning from human feedback is difficult because methods\nlike [RLHF](https://arxiv.org/abs/2203.02155) are complex and costly to run.\nAlpacaFarm is a simulator that enables research and development on learning from feedback at a fraction of the usual\ncost, promoting accessible research on instruction following and alignment.\n\nPlease read our [paper](https://arxiv.org/abs/2305.14387)\nand [blog post](https://crfm.stanford.edu/2023/05/22/alpaca-farm.html) for details on our research findings.\n\nThis repo contains code for\n\n- [simulating preference feedback from language models such as GPT-4](#simulating-pairwise-preference)\n- [automated evaluation for instruction-following models](#running-automatic-evaluation)\n- [validated reference implementations of baseline methods such as PPO and best-of-n](#running-reference-methods)\n\nThe data needed to run our code is hosted on HuggingFace: \u003chttps://huggingface.co/datasets/tatsu-lab/alpaca_farm\u003e.\n\n**Usage and License Notices**: AlpacaFarm is intended and licensed for research use only.\nThe dataset is CC BY NC 4.0 (allowing only non-commercial use) and models trained using the dataset should not be used\noutside of research purposes.\nThe weight diff is also CC BY NC 4.0 (allowing only non-commercial use).\n\n## The AlpacaFarm\n\n\u003cbr\u003e\n\u003cp style=\"text-align:center;\"\u003e\n  \u003cimg style=\"max-width:70%; height:auto;\" src=\"./assets/fig1.jpg\" alt=\"Workflow\"\u003e\n\u003c/p\u003e\n\nInstruction-following models are typically developed in 3 steps\n\n1. Supervised fine-tuning with demonstrations\n2. Learning from human feedback; usually pairwise preferences\n3. Human evaluation with interaction\n\nThe goal of AlpacaFarm is to provide three key components that tackles steps 2 and 3:\nLow-cost simulation of pairwise feedback from API models (e.g. GPT-4, ChatGPT), automated evaluations for methods\ndevelopment, and reference implementations of\nlearning algorithms for comparison and modification.\n\n## Installation\n\nTo install the stable release, run\n\n```bash\npip install alpaca-farm\n```\n\nTo install from the latest commit on `main` branch, run\n\n```bash\npip install git+https://github.com/tatsu-lab/alpaca_farm.git\n```\n\nTo enable FlashAttention and other optimizations, install\nthe [`flash-attn`](https://github.com/HazyResearch/flash-attention) and [`apex`](https://github.com/NVIDIA/apex)\npackages.\n\n## Simulating pairwise preference\n\n**Notebook\nexample:** [![Using](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tatsu-lab/alpaca_farm/blob/main/examples/auto_annotations.ipynb)\n\nFor all the evaluation and annotations we use [**AlpacaEval**](https://github.com/tatsu-lab/alpaca_eval/tree/main#making-a-new-evaluator) with our pool of automatic annotators and additional noise to simulate the variance of human annotations.\n\nTo get started, set the environment variable `OPENAI_API_KEY` to your OpenAI API key, and (optionally) `OPENAI_ORG` to\nthe\norganization ID.\nYou can do this by running\n\n```bash\nexport OPENAI_API_KEY=\"sk...\"\n```\n\nTo annotate the pairs of outputs of your model use the following code.\nFor more details or functions to use if you have outputs in different formats refer to\nthe [example notebook](https://github.com/tatsu-lab/alpaca_farm/blob/main/examples/auto_annotations.ipynb).\n\n```python\nfrom alpaca_farm.auto_annotations import PairwiseAutoAnnotator\nimport json\n\n# load some data\nwith open(\"examples/data/outputs_pairs.json\") as f:\n    outputs_pairs = json.load(f)[:6]\nprint(outputs_pairs[-1:])\n# [{'instruction': 'If you could help me write an email to my friends inviting them to dinner on Friday, it would be greatly appreciated.',\n#   'input': '',\n#   'output_1': \"Dear Friends, \\r\\n\\r\\nI hope this message finds you well. I'm excited to invite you to dinner on Friday. We'll meet at 7:00 PM at [location]. I look forward to seeing you there. \\r\\n\\r\\nBest,\\r\\n[Name]\",\n#   'output_2': \"Hey everyone! \\n\\nI'm hosting a dinner party this Friday night and I'd love for all of you to come over. We'll have a delicious spread of food and some great conversations. \\n\\nLet me know if you can make it - I'd love to see you all there!\\n\\nCheers,\\n[Your Name]\"}]\n\nannotator = PairwiseAutoAnnotator()\nannotated = annotator.annotate_pairs(outputs_pairs)\n\nprint(annotated[-1:])\n# [{'instruction': 'If you could help me write an email to my friends inviting them to dinner on Friday, it would be greatly appreciated.', \n# 'input': '', \n# 'output_1': \"Dear Friends, \\r\\n\\r\\nI hope this message finds you well. I'm excited to invite you to dinner on Friday. We'll meet at 7:00 PM at [location]. I look forward to seeing you there. \\r\\n\\r\\nBest,\\r\\n[Name]\", \n# 'output_2': \"Hey everyone! \\n\\nI'm hosting a dinner party this Friday night and I'd love for all of you to come over. We'll have a delicious spread of food and some great conversations. \\n\\nLet me know if you can make it - I'd love to see you all there!\\n\\nCheers,\\n[Your Name]\",\n# 'annotator': 'chatgpt_2', \n# 'preference': 2}]\n```\n\nIf instead of pairs you have a list of sampled outputs, you can use the following.\n\n```python\nmultisample_outputs = [dict(instruction=\"repeat the following\", input=\"yes\", output=[\"yes\", \"no\", \"maybe\", \"repeat\"])]\nprint(annotator.annotate_samples(multisample_outputs))\n# [{'sample_id': 0, \n#   'instruction': 'repeat the following', \n#   'input': 'yes', \n#   'output_1': 'yes', \n#   'output_2': 'maybe', \n#   'annotator': 'chatgpt_2', \n#   'preference': 1}]\n```\n\n## Running automatic evaluation\n\nFor all the evaluation we use [**AlpacaEval**](https://github.com/tatsu-lab/alpaca_eval/tree/main#making-a-new-evaluator) with our pool of automatic annotators. \n\nTo get started, set the environment variable OPENAI_API_KEY to your OpenAI API key, and (optionally) OPENAI_ORG to the\norganization ID. You can do this by running\n\n```bash\nexport OPENAI_API_KEY=\"sk...\"\n```\n\nThe easiest to add your model to the Alpaca Leaderboard is to run the following code, which only requires having outputs\nfor your model on our eval data.\n\n```python\nfrom alpaca_farm.auto_annotations import alpaca_leaderboard\nimport datasets\n\n# predict on Alpaca eval data\nalpaca_eval_data = datasets.load_dataset(\"tatsu-lab/alpaca_farm\", \"alpaca_farm_evaluation\")[\"eval\"]\n...  # use the data to get outputs for your model and save it\npath_to_outputs = \"examples/data/eval_gpt-3.5-turbo-0301.json\"\n# outputs should be a list of json as such:\n# [{'instruction': 'What are the names of some famous actors that started their careers on Broadway?', 'input': '', 'output': 'Some famous actors that started their careers on Broadway are Hugh Jackman, Meryl Streep, Denzel Washington, Audra McDonald, and Lin-Manuel Miranda.', 'generator': 'gpt-3.5-turbo-0301', 'dataset': 'helpful_base', 'datasplit': 'eval'},\n# ...]\n\nalpaca_leaderboard(path_to_outputs, name=\"My fancy model\")\n#                               win_rate  standard_error  n_total  avg_length\n# gpt35_turbo_instruct             81.71            1.33      801        1018\n# alpaca-farm-ppo-sim-gpt4-20k     44.10            1.74      805         511\n# My fancy model                   41.54            2.01      597         327\n# alpaca-farm-ppo-human            41.24            1.73      805         803\n# alpaca-7b                        26.46            1.54      805         396\n# text_davinci_001                 15.17            1.24      804         296\n```\n\n## Running reference methods\n\nWe provide reference implementations of several methods for learning from pairwise feedback.\nExample code to run these methods can be found in the `examples/` directory.\nThis includes [supervised fine-tuning](examples/supervised.py), [reward modeding](examples/reward_modeling.py)\n, [RLHF with PPO](examples/rlhf_ppo.py), [best-of-n decoding](examples/best_of_n.py) and more.\n\nBelow we give example commands for reproducing the model artifacts in our paper. Notes:\n\n- All training code are tested with FlashAttention enabled on a machine with 8 80GB A100 GPUs.\n- Best-of-n decoding was tested with a single 80GB GPU.\n- Supervised fine-tuning and reward modeling can fit on 4 80GB A100 GPUs, while PPO training currently requires at least\n  8\n  80GB GPUs.\n- Before running the code below, make sure to convert your LLaMA checkpoint and tokenizer into HuggingFace format and\n  store it at `\u003cyour_path_to_hf_converted_llama_ckpt_and_tokenizer\u003e`.\n\n### Supervised fine-tuning (SFT)\n\nTo replicate our SFT10k model fine-tuned from LLaMA in the paper, run\n\n```bash\nbash examples/scripts/sft.sh \\\n  \u003cyour_output_dir_for_sft10k\u003e \\\n  \u003cyour_wandb_run_name\u003e \\\n  \u003cyour_path_to_hf_converted_llama_ckpt_and_tokenizer\u003e\n```\n\nThe SFT10k model will be saved at `\u003cyour_output_dir\u003e`, and the name of the wandb run will be `\u003cyour_wandb_run_name\u003e`.\n\n### Reward modeling\n\nTo replicate our reward models trained in the paper, run\n\n```bash\nbash examples/scripts/reward_modeling.sh \\\n  \u003cyour_output_dir_for_reward_model\u003e \\\n  \u003cyour_wandb_run_name\u003e \\\n  \u003cyour_output_dir_for_sft10k\u003e \\\n  \u003cpreference_dataset_name\u003e\n```\n\nSet `\u003cpreference_dataset_name\u003e` to `\"alpaca_noisy_multi_preference\"` for simulated preference reward model, and\n`\"alpaca_human_preference\"` for human preference reward model.\n\n### RLHF with PPO\n\nTo replicate our RLHF PPO model trained with simulated reward model in the paper, run\n\n```bash\nbash examples/scripts/rlhf_ppo.sh \\\n  \u003cyour_output_dir_for_ppo\u003e \\\n  \u003cyour_wandb_run_name\u003e \\\n  \u003cyour_output_dir_for_reward_model\u003e \\\n  \u003cyour_output_dir_for_sft10k\u003e \\\n  \u003ckl_coef\u003e\n```\n\n`\u003cyour_output_dir_for_reward_model\u003e` should point to either simulated reward model or human reward model trained\naccording\nto the previous step.\nNote the KL penalty coefficient for human reward PPO is much larger than for simulated PPO.\nSet `\u003ckl_coef\u003e` to `0.0067` for simulated PPO, and `0.02` for human PPO to recover our original results.\nPerformance of the PPO model is typically much better than SFT at 20-80 PPO steps (less than 4 passes through the entire\nset of instructions) and starts to decay with more PPO steps.\n\n### Best-of-n decoding\n\nTo replicate our best-of-n inference-time decoding results for the AlpacaFarm evaluation suite, run\n\n```bash\npython examples/best_of_n.py \\\n  --task \"run_best_of_n\" \\\n  --decoder_name_or_path \u003cyour_output_dir_for_decoder\u003e \\  # Can be SFT model or even PPO tuned model.\n  --scorer_name_or_path \u003cyour_output_dir_for_reward_model\u003e \\\n  --num_return_sequences 16 \\  # This is the n in best-of-n.\n  --per_device_batch_size 4 \\  # Reduce this if you don't have enough memory.\n  --split \"eval\" \\\n  --mixed_precision \"bf16\" \\\n  --tf32 True \\\n  --flash_attn True \\\n  --output_path \u003cyour_output_path_to_store_samples\u003e\n```\n\nYou can then use the generated samples at `\u003cyour_output_path_to_store_samples\u003e` directly with our automated evaluation.\n\n### Expert Iteration\n\nTo replicate our expert iteration results for the AlpacaFarm evaluation suite, first produce best-of-n samples. Run\n\n```bash\npython examples/best_of_n.py \\\n  --task \"run_best_of_n\" \\\n  --decoder_name_or_path \u003cyour_output_dir_for_decoder\u003e \\  # SFT10k model.\n  --scorer_name_or_path \u003cyour_output_dir_for_reward_model\u003e \\\n  --num_return_sequences 16 \\  # This is the n in best-of-n.\n  --per_device_batch_size 4 \\  # Reduce this if you don't have enough memory.\n  --split \"unlabeled\" \\\n  --mixed_precision \"bf16\" \\\n  --tf32 True \\\n  --flash_attn True \\\n  --output_path '\u003cyour_output_dir_for_expiter_data\u003e/best_of_n_samples.json'\n```\n\nThen perform supervised fine-tuning from the SFT10k checkpoint with the best-of-n samples\n\n```bash\nbash examples/scripts/expiter.sh \\\n  \u003cyour_output_dir_for_expiter\u003e \\\n  \u003cyour_wandb_run_name\u003e \\\n  \u003cyour_output_dir_for_sft10k\u003e \\\n  \u003cyour_output_dir_for_expiter_data\u003e\n```\n\n### Quark\n\nTo replicate our Quark results for the AlpacaFarm evaluation suite, run\n\n```bash\nbash examples/scripts/rlhf_quark.sh \\\n  \u003cyour_output_dir_for_quark\u003e \\\n  \u003cyour_wandb_run_name\u003e \\\n  \u003cyour_output_dir_for_reward_model\u003e \\\n  \u003cyour_output_dir_for_sft10k\u003e \\\n  \u003ckl_coef\u003e\n```\n\n### [Direct Preference Optimization (DPO)](https://arxiv.org/abs/2305.18290)\n\nTo replicate our DPO results for the AlpacaFarm evaluation suite, run\n\n```bash\nbash examples/scripts/dpo.sh \\\n  \u003cyour_output_dir_for_dpo\u003e \\\n  \u003cyour_wandb_run_name\u003e \\\n  \u003cyour_output_dir_for_sft10k\u003e\n```\n\n### OpenAI models\n\nTo run the OpenAI reference models with our prompts and decoding hyperparameters, run\n\n```bash\npython examples/oai_baselines.py \\\n  --model_name \u003coai_model_name\u003e \\\n  --save_path \u003csave_path\u003e \n```\n\nYou can then use the generated samples at `\u003csave_path\u003e` directly with our automated evaluation.\n\n## Downloading pre-tuned AlpacaFarm models\n\nWe provide model checkpoints for reward models and all our reference methods, listed in Table 2 of\nour [paper](https://arxiv.org/abs/2305.14387). Concretely, we tune each reference method in AlpacaFarm simulation and on\nhuman preference data and release both versions. The current list of models\n(available [here](https://huggingface.co/tatsu-lab)) includes:\n\n- `sft10k`, the supervised learning base model that we collect preference data with.\n- `reward-model-sim`, the reward model trained on AlpacaFarm preference data.\n- `reward-model-human`, the reward model trained on human preference data.\n- `ppo-sim`, the best PPO checkpoint trained in simulation.\n- `ppo-human`, the best PPO checkpoint trained on human data.\n- `expiter-sim`, the best expert iteration checkpoint trained in simulation.\n- `expiter-human`, the best expert iteration checkpoint trained on human data.\n- `feedme-sim`, the FeedME method trained on simulated preferences.\n- `feedme-human`, the FeedME method trained on human preferences.\n- `reward-condition-sim`, the reward conditioning method trained on simulated preferences.\n\nTo download and recover these checkpoints, first make sure to have a LLaMA-7B\ncheckpoint [converted into the Hugging Face format](https://huggingface.co/docs/transformers/main/model_doc/llama)\n**with transformers\u003e=4.29.2**.\nThen, run the following to download all AlpacaFarm models:\n\n```\npython -m pretrained_models.recover_model_weights \\\n  --llama-7b-hf-dir \u003cyour_path_to_hf_converted_llama_ckpt_and_tokenizer\u003e \\\n  --alpaca-farm-model-name all\n```\n\nOr, specify a particular model name to download just that model:\n\n```\npython -m pretrained_models.recover_model_weights \\\n  --llama-7b-hf-dir \u003cyour_path_to_hf_converted_llama_ckpt_and_tokenizer\u003e \\\n  --alpaca-farm-model-name \u003cone_of_the_model_names_from_above\u003e \\\n  --models-save-dir \u003cdir_to_save_all_models\u003e\n```\n\nTo download either of the reward models individually, you'll need to have `sft10k` downloaded first\nto `\u003cdir_to_save_all_models\u003e`.\n\n## Citation\n\nPlease consider citing our work if you use the data or code in this repo.\n\n```\n@misc{dubois2023alpacafarm,\n      title={AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback}, \n      author={Yann Dubois and Xuechen Li and Rohan Taori and Tianyi Zhang and Ishaan Gulrajani and Jimmy Ba and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto},\n      year={2023},\n      eprint={2305.14387},\n      archivePrefix={arXiv},\n      primaryClass={cs.LG}\n}\n```\n\nIf you use `alpaca-farm\u003e=0.2.0` make sure to specify that the annotator changed (as `text-davinci-003` is depreciated). The preferences and win-rates are now from AlpacaEval 1 and are not comparable to the numbers from our paper. You can cite AlpacaEval as:\n\n```\n@misc{alpaca_eval,\n  author = {Xuechen Li and Tianyi Zhang and Yann Dubois and Rohan Taori and Ishaan Gulrajani and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto },\n  title = {AlpacaEval: An Automatic Evaluator of Instruction-following Models},\n  year = {2023},\n  publisher = {GitHub},\n  journal = {GitHub repository},\n  howpublished = {\\url{https://github.com/tatsu-lab/alpaca_eval}}\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftatsu-lab%2Falpaca_farm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftatsu-lab%2Falpaca_farm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftatsu-lab%2Falpaca_farm/lists"}