{"id":19539613,"url":"https://github.com/wangclnlp/deepspeed-chat-extension","last_synced_at":"2025-04-26T16:31:45.911Z","repository":{"id":216655824,"uuid":"731911858","full_name":"wangclnlp/DeepSpeed-Chat-Extension","owner":"wangclnlp","description":"This repo contains some extensions of deepspeed-chat for fine-tuning LLMs (SFT+RLHF).","archived":false,"fork":false,"pushed_at":"2024-07-02T15:30:57.000Z","size":12251,"stargazers_count":18,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-04T15:21:25.982Z","etag":null,"topics":["deepspeed","llama","llm","rlhf","sft"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wangclnlp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-15T07:10:36.000Z","updated_at":"2025-02-01T12:18:05.000Z","dependencies_parsed_at":"2024-01-11T18:57:43.624Z","dependency_job_id":"f6b6e1ce-d98f-4670-a996-54ad1fd8f774","html_url":"https://github.com/wangclnlp/DeepSpeed-Chat-Extension","commit_stats":null,"previous_names":["wangclnlp/deepspeed-chat-extension"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wangclnlp%2FDeepSpeed-Chat-Extension","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wangclnlp%2FDeepSpeed-Chat-Extension/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wangclnlp%2FDeepSpeed-Chat-Extension/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wangclnlp%2FDeepSpeed-Chat-Extension/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wangclnlp","download_url":"https://codeload.github.com/wangclnlp/DeepSpeed-Chat-Extension/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251017526,"owners_count":21523599,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deepspeed","llama","llm","rlhf","sft"],"created_at":"2024-11-11T02:42:37.540Z","updated_at":"2025-04-26T16:31:40.893Z","avatar_url":"https://github.com/wangclnlp.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"We have edited the code of project [DeepSpeed-Chat](https://github.com/microsoft/DeepSpeedExamples/tree/master/applications/DeepSpeed-Chat) to support many new features as shown below.\n\n# Our New Features🎉🎉🎉\n- We propose a hybrid alignment training to improve the LLM ([./examples/hybrid_alignment_training](https://github.com/wangclnlp/DeepSpeed-Chat-Extension/tree/main/examples/hybrid_alignment_training)).\n- Add extra loss for RLHF in step3 like SFT loss and pre-trained loss ([./examples/add_extra_loss_for_rlhf](./examples/add_extra_loss_for_rlhf)).\n- Support [DPO](https://arxiv.org/abs/2305.18290) as step2 ([./examples/dpo](./examples/dpo)).\n- Implement [ESRL](https://arxiv.org/abs/2308.02223) features to train efficiently in step3 ([./examples/esrl](./examples/esrl)).\n- Support COMET model(s) as reward model(s) in step3 RLHF ([./examples/rlhf_with_comet_reward](./examples/rlhf_with_comet_reward)).\n- Support using scores instead of pairwise data only to train reward models directly ([./examples/training_reward_with_scores](./examples/training_reward_with_scores)).\n\nMore details in [./examples](./examples).\n\n# Installation\n\nYou can use anaconda/miniconda to install packages needed for this project.\n\n```bash\nconda env create -f conda-env.yml\nconda activate dschat\npip install -r requirements.txt\n```\n\n# Training Models\n\n## Step1 Supervised Fine-tuning (SFT)\n\n```bash\nbash scripts/sft.sh\n```\n\n## Step2 Reward Model Fine-tuning\n\n```bash\nbash scripts/reward.sh\n```\n\n## Step2 Direct Pereference Optimization (DPO)\n\n```bash\nbash examples/dpo/train.sh\n```\n\n## Step3 Reinforcement Learning from Human Feedback (RLHF)\n\n```bash\nbash scripts/rlhf.sh\n```\n\n# Supported Models\n\n| Model | Model size |\n|:---:|:---:|\n| Baichuan | 7B/13B |\n| Baichuan2 | 7B/13B |\n| LLaMA | 7B/13B/33B/65B |\n| LLaMA-2 | 7B/13B/70B |\n| Yi | 6B/34B |\n\n# Format of the Dataset\n\n## SFT\n\nThe dataset for SFT should be `txt` files including `train.txt` and `test.txt`  with `sft` in path such as `/your/path/to/sft_dataset/train.txt`, containing a json string each line as example below.\n\nExample:\n\n```\n{\"instruction\": \"User: Your task is to ... \\nAssistant: \", \"input\": \"...\", \"output\": \"...\"}\n...\n```\n\n## SFT with Multi-turn History\n\nWe also support sft training with multi-turn dialogues. The corresponding dataset also contains a json string on each line, as shown in the example below.\n\nExample:\n\n```\n{\n \"instruction\": \"User: Your task is to ... \\nAssistant: \",\n \"input\": \"...\",\n \"output\": \"...\",\n \"history\": [\n              [\"user instruction in the first round (optional)\", \"model response in the first round (optional)\"],\n              [\"user instruction in the second round (optional)\", \"model response in the second round (optional)\"],\n              ...\n            ]\n}\n...\n```\n\n## Reward/DPO\n\nThe dataset for Reward/DPO should be parquet files including `train.parquet` and `test.parquet` with `reward` in path such as `/your/path/to/reward_dataset/train.parquet`, containing four keys each entry as example below.\n\nExample:\n\n| prompt | response | chosen | rejected |\n|:---:|:---:|:---:|:---:|\n| User: What are some of the challenges with usi... | Some of the challenges with using machine lear... | Some of the challenges with using machine lear... | Machine learning is a very powerful tool. |\n| User: Looking for an essay by a contemporary m... | I believe you're thinking of Bernard-Henri Lévy. | I believe you're thinking of Bernard-Henri Lévy. | Laclau maybe? |\n| ... | ... | ... | ... |\n\n## RLHF\n\nSame as SFT, except for `rlhf` in path such as `/your/path/to/rlhf_dataset/train.txt`.\n\n# Inference\n\nYou can use [this](rlhf_llama/deepspeed_chat/training/step1_supervised_finetuning/predict.py) python script for inference as shown in [`./scripts/predict.sh`](./scripts/predict.sh) in which the input should be in format of `{Input} ||| {None/Reference}` while output would be `{Input} ||| {ModelOutput} ||| {None/Reference}` as example below.\n\nExample:\n\ninput.txt\n```\nUser: What are the names of some famous actors ...\\nAssistant: ||| Some famous ...\nUser: ...                                                      ||| None\n...                                                            ||| ...\n```\n\noutput.txt\n```\nUser: What are the names of some famous actors ...\\nAssistant: ||| 1. Denzel Washington ... ||| Some famous ...\nUser: ...                                                      ||| ...                      ||| None\n...                                                            ||| ...                      ||| ...\n```\n\n\n# Last but Not Least\n\nThanks to the [DeepSpeed-Chat](https://github.com/microsoft/DeepSpeedExamples/tree/master/applications/DeepSpeed-Chat) project and its contributors❤️❤️❤️!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwangclnlp%2Fdeepspeed-chat-extension","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwangclnlp%2Fdeepspeed-chat-extension","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwangclnlp%2Fdeepspeed-chat-extension/lists"}