{"id":20298115,"url":"https://github.com/hkust-nlp/deita","last_synced_at":"2025-05-15T17:08:03.080Z","repository":{"id":214134906,"uuid":"710119416","full_name":"hkust-nlp/deita","owner":"hkust-nlp","description":"Deita: Data-Efficient Instruction Tuning for Alignment [ICLR2024]","archived":false,"fork":false,"pushed_at":"2024-12-09T02:45:47.000Z","size":246,"stargazers_count":546,"open_issues_count":5,"forks_count":29,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-04-07T23:04:50.690Z","etag":null,"topics":["alignment","data-centric","instruction-tuning","large-language-models"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hkust-nlp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-26T04:00:47.000Z","updated_at":"2025-04-03T07:44:07.000Z","dependencies_parsed_at":"2024-01-01T04:39:51.978Z","dependency_job_id":"ef1253c9-774e-46a2-ba6a-7f6c70961e60","html_url":"https://github.com/hkust-nlp/deita","commit_stats":null,"previous_names":["hkust-nlp/deita"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hkust-nlp%2Fdeita","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hkust-nlp%2Fdeita/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hkust-nlp%2Fdeita/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hkust-nlp%2Fdeita/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hkust-nlp","download_url":"https://codeload.github.com/hkust-nlp/deita/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254384988,"owners_count":22062422,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["alignment","data-centric","instruction-tuning","large-language-models"],"created_at":"2024-11-14T16:02:11.685Z","updated_at":"2025-05-15T17:07:58.072Z","avatar_url":"https://github.com/hkust-nlp.png","language":"Python","funding_links":[],"categories":["A01_文本生成_文本对话","Python","SFT Statistics"],"sub_categories":["大语言对话模型及数据","General SFT"],"readme":"# Deita\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./assets/logo-final.png\" width=\"600\"\u003e\n\u003c/p\u003e\n\n\n\u003cp align=\"center\"\u003e\n  🤗 \u003ca href=\"https://huggingface.co/collections/hkust-nlp/deita-6569c198c174808d94cf5bd4\"\u003eHF Repo\u003c/a\u003e\u0026nbsp;\u0026nbsp;\u0026nbsp;\n  📄 \u003ca href=\"https://arxiv.org/abs/2312.15685\"\u003ePaper\u003c/a\u003e\u0026nbsp;\u0026nbsp;\u0026nbsp;\n  📚 \u003ca href=\"https://huggingface.co/datasets/hkust-nlp/deita-6k-v0\"\u003e6K Data\u003c/a\u003e\u0026nbsp;\u0026nbsp;\u0026nbsp;\n  📚 \u003ca href=\"https://huggingface.co/datasets/hkust-nlp/deita-10k-v0\"\u003e10K Data\u003c/a\u003e\n\u003c/p\u003e\n\n\nWelcome to Deita (**D**ata-**E**fficient **I**nstruction **T**uning for **A**lignment) Project! \n\nWe will continue to update, please stay tuned!\n\n\n## What is Deita?\nDeita is an open-sourced project designed to facilitate **Automatic Data Selection** for instruction tuning in Large Language Models (LLMs).\n\nIt includes:\n- **Open-sourced Toolkits** for automatic data selection in instruction tuning\n- **Deita Datasets**: A series of extremely *lightweight*, high-quality alignment SFT data. We release 6k-sized and 10k-sized datasets in the first release\n- **Deita Models**: A series of powerful models on par with SOTA chat LLMs with an extremely efficient instruction tuning Process. Deita models can be obained by training with 10x less instruction tuning data compared with other SOTA LLMs\n\n## News\n- :fire: [03/2024] Our datasets have been used by Huggingface to creat the [Zephyr Gemma Model](https://huggingface.co/collections/HuggingFaceH4/zephyr-7b-gemma-65e1fd82d26b426e3e63d956).\n- 📄 [01/2024] Deita paper [What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning](https://arxiv.org/abs/2312.15685) has been accepted by ICLR2024!\n- :fire: [01/2024] [Deita pipelines](#deita-pipelines) have been released! With one line code and configurations, a high-quality data subset for alignment can be selected.\n- 📚 [01/2024] Our scorer datasets [deita-complexity-scorer-data](https://huggingface.co/datasets/hkust-nlp/deita-complexity-scorer-data) and [deita-quality-scorer-data](https://huggingface.co/datasets/hkust-nlp/deita-quality-scorer-data) have been released.\n- :fire: [12/2023] We release the first collection of the Deita resources [here](https://huggingface.co/collections/hkust-nlp/deita-6569c198c174808d94cf5bd4), which include a series of extremely lightweight, effective sft datasets, the data complexity/quality scorer models, as well as the resulted deita chat models. \n\n## Performance\n:bell: Still curious about how far a small amount of high-quality data can lead LLMs? \n\nDeita may provide an answer for you:\n\n**🔦 Highlights**\n| Model                                          | Align        | Data Size  | MT-Bench | AlpacaEval(%) |\n|------------------------------------------------|--------------|------------|----------|---------------|\n| Zephyr-7B-sft                                  | SFT          | 200K       | 5.32     | 75.12         |\n| $\\text{Zephyr-7B-}\\beta$                      | SFT + DPO    | 200K SFT + 60K DPO | 7.34     | 90.60         |\n| OpenChat-3.5                                   | C-RLFT | \u003e\u003e 70K C-RLFT | 7.81     | 88.51         |\n| Starling-7B                                    | C-RLFT + APA | \u003e\u003e 70K C-RLFT + 183K APA | 8.09     | 91.99         |\n| Tulu-2-13B                                     | SFT          | 326K       | 6.70     | 78.90         |\n| Tulu-2-13B+DPO                                 | SFT + DPO    | 326K SFT + 60K DPO | 7.00     | 89.50         |\n| LLaMA2-13B-Chat                                | SFT + PPO    | --         | 6.65     | 81.09         |\n| WizardLM-13B-v1.2                              | SFT          | \u003e70K       | 7.09     | 89.17         |\n| Vicuna-13B-v1.5                                | SFT          | \u003e125K      | 6.57    | 78.80         |\n| DEITA-7B-v1.0 (6K)          | SFT          | 6K       |   7.22   |    80.78      |\n| DEITA-7B-v1.0-sft            | SFT          | 10K        | 7.32     | 81.67         |\n| DEITA-7B-v1.0 | SFT + DPO    | 6K SFT + 10K DPO | 7.55     | 90.06         |\n\nDEITA models are based on Mistral-7B-v0.1. :fire: \n\nPlease refer to [this table](#chart\\_with\\_upwards\\_trend-full-evaluations) for full evaluations including Open LLM Leaderboard as well, which includes DEITA models with LLaMA base models and comparisons with other data selection approaches.\n\n\n\n## :chart_with_upwards_trend: Full Evaluations\n\n\u003cdetails\u003e\n  \u003csummary\u003eSee full evaluations\u003c/summary\u003e\n\n| Model                                          | Align     | Data Size  | MT-Bench | AlpacaEval(%) | OpenLLM (Avg.) |\n|------------------------------------------------|-----------|------------|----------|---------------|----------------|\n| **Proprietary Models**                         |           |            |          |               |                |\n| GPT-4-Turbo                                    | ?         | --         | 9.32     | 97.70         | --             |\n| GPT-4                                          | SFT + PPO | --         | 8.99     | 95.03         | --             |\n| Claude-2                                       | SFT + PPO | --         | 8.06     | 91.36         | --             |\n| GPT-3.5-turbo                                  | SFT + PPO | --         | 7.94     | 89.37         | --             |\n| **Open-sourced Models based on LLaMA-1-13B**   |           |            |          |               |                |\n| LIMA                                           | SFT       | 1K SFT        | 4.29     | 41.98         | 59.82          |\n| WizardLM-13B                                   | SFT       | 70K SFT       | 6.35     | 75.31         | 58.96          |\n| Vicuna-13B-v1.3                                | SFT       | 125K SFT      | 6.39     | 82.11         | 60.01          |\n| Random                                         | SFT       | 10K SFT       | 6.03     | 71.52         | 60.14          |\n| DEITA-LLaMA1-13B-v1.0-sft                           | SFT       | 10K SFT       | 6.60     | 78.01         | 64.27          |\n| **Open-sourced Models based on LLaMA-2-13B**   |           |            |          |               |                |\n| Tulu-2-13B                                     | SFT       | 326K SFT      | 6.70     | 78.90         | --             |\n| Tulu-2-13B+DPO                                 | SFT + DPO | 326K SFT + 60K DPO | 7.00     | 89.50         | --             |\n| LLaMA2-13B-Chat                                | SFT + PPO | --         | 6.65     | 81.09         | --             |\n| WizardLM-13B-v1.2                              | SFT          | \u003e70K SFT      | 7.09     | 89.17         | --             |\n| Vicuna-13B-v1.5                                | SFT       | 125K SFT      | 6.57     | 78.80         | 61.63          |\n| Random                                         | SFT       | 10K SFT       | 5.78     | 65.19         | 61.32          |\n| DEITA-LLaMA2-13B-v1.0-sft                           | SFT       | 10K SFT       | 6.79     | 81.09         | 62.71          |\n| **Open-sourced Models based on Mistral-7B**    |           |            |          |               |                |\n| Mistral-7B-Instruct-v0.1                       | --        | --         | 6.84     | 69.65         | 60.45          |\n| Zephyr-7B-sft                                  | SFT       | 200K SFT      | 5.32     | 75.12         | 60.93          |\n| $\\text{Zephyr-7B-}\\beta$                       | SFT + DPO | 200K SFT + 60K DPO | 7.34     | 90.60         | 66.36          |\n| OpenChat-3.5                                   | C-RLFT | \u003e\u003e 70K C-RLFT | 7.81     | 88.51         | --           |\n| Starling-7B                                    | C-RLFT + APA | \u003e\u003e70K C-RLFT + 183K APA | 8.09     | 91.99         | --            |\n| Random                                         | SFT       | 10K SFT       | 5.89     | 56.90         | 61.72          |\n| DEITA-7B-v1.0-sft (6K)                           | SFT       | 6K SFT       | 7.22     | 80.78         | 64.94          |\n| DEITA-7B-v1.0-sft (10K)                  | SFT       | 10K SFT       | 7.32     | 81.67         | 64.00          |\n| DEITA-7B-v1.0             | SFT + DPO | 6K SFT + 10K DPO   | 7.55     | 90.06         | 69.86          |\n\n\n\u003c/details\u003e\n\n## :rocket: Deita Resources\n\n| Resource                                       | Link     | License  |\n|------------------------------------------------|-----------|------------|\n| **Deita Datasets**                                   |           |            |\n| deita-6k-v0                                    | [:hugs: HF Repo](https://huggingface.co/datasets/hkust-nlp/deita-6k-v0)          | [MIT License](https://opensource.org/license/mit/) |\n| deita-10k-v0                                    | [:hugs: HF Repo](https://huggingface.co/datasets/hkust-nlp/deita-10k-v0)          | [MIT License](https://opensource.org/license/mit/) |\n| deita-complexity-scorer-data                                    | [:hugs: HF Repo](https://huggingface.co/datasets/hkust-nlp/deita-complexity-scorer-data)          | [MIT License](https://opensource.org/license/mit/) |\n| deita-quality-scorer-data                                    | [:hugs: HF Repo](https://huggingface.co/datasets/hkust-nlp/deita-quality-scorer-data)          | [MIT License](https://opensource.org/license/mit/) |\n| deita-redundant-pool (100K)                                    | [:hugs: HF Repo](https://huggingface.co/datasets/hkust-nlp/deita-redundant-pool-data)          | [MIT License](https://opensource.org/license/mit/) |\n| deita-sota-pool (300K)                                    | [:hugs: HF Repo](https://huggingface.co/datasets/AndrewZeng/deita_sota_pool)          | [MIT License](https://opensource.org/license/mit/) |\n| **Scorers**                                   |           |             |\n|  deita-complexity-scorer                      | [:hugs: HF Repo](https://huggingface.co/hkust-nlp/deita-complexity-scorer)          | [LLaMA License](https://ai.meta.com/resources/models-and-libraries/llama-downloads/)|\n|  deita-quality-scorer               | [:hugs: HF Repo](https://huggingface.co/hkust-nlp/deita-quality-scorer)          | [LLaMA License](https://ai.meta.com/resources/models-and-libraries/llama-downloads/)|\n| **Deita Models**                                   |           |             |\n| DEITA-7B-v1.0-sft                | [:hugs: HF Repo](https://huggingface.co/hkust-nlp/deita-7b-v1.0-sft)           | [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)             |\n| DEITA-7B-v1.0                | [:hugs: HF Repo](https://huggingface.co/hkust-nlp/deita-7B-v1.0)           | [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)             |\n| DEITA-LLaMA2-13B-v1.0-sft         | [:hugs: HF Repo](https://huggingface.co/hkust-nlp/deita-llama2-13b-v1.0-sft)           |  [LLaMA 2 License](https://ai.meta.com/resources/models-and-libraries/llama-downloads/)           |\n| DEITA-LLaMA1-13B-v1.0-sft          | [:hugs: HF Repo](https://huggingface.co/hkust-nlp/deita-llama1-13b-v1.0-sft)          |  [LLaMA License](https://ai.meta.com/resources/models-and-libraries/llama-downloads/)           |\n\n## :running_man: How to start?\n\n\n### Installation\n```bash\n  git clone https://github.com/hkust-nlp/deita.git\n  cd deita\n  pip install -e .\n```\n\n### Data Sample Scoring\n\nIf you wish to assess the **quality** of a response for a single sample, you can follow these steps:\n```python\nfrom deita.selection.scorer import Llama_Scorer\n\nmodel_name_or_path = \"hkust-nlp/deita-quality-scorer\"\n\nscorer = Llama_Scorer(model_name_or_path)\n\n# example input\ninput_text = \"word to describe UI with helpful tooltips\" # Example Input\noutput_text = \"User-friendly or intuitive UI\" # Example Output\nquality_score = scorer.infer_quality(input_text, output_text)\n\nprint(quality_score)\n# 2.0230105920381902\n```\n\nDeita also supports VLLM for faster inference. If you want to use VLLM for inference,\n\n```bash\npip install vllm\n```\n\nAnd set ```is_vllm = True``` when initilizing scorer\n\n```python\nscorer = Llama_Scorer(model_name_or_path, is_vllm = True)\n```\n\nTo assess other dimensions of data samples, please refer to the ```examples/scoring```\n\n### Deita Pipelines\n\nYou can use deita pipelines to perform a variety of operations on the dataset with only one line code and configurations.\n\n- **Dataset Scoring**\n\n```python\nfrom deita.pipeline import Pipeline\n\npipeline = Pipeline(\"score_pipeline\", \n                    data_path = args.data_path,   # json file with sharegpt format\n                    scorer = args.scorer,   # [mistral, llama]\n                    scorer_name_or_path = args.scorer_name_or_path,  # scorer name or path e.g. hkust-nlp/deita-complexity-scorer\n                    is_vllm = args.is_vllm,  # launch with vllm [True, False]\n                    score_type = args.score_type, # [complexity, quality]\n                    output_path = args.output_path)  # output path (json format)\n\npipeline.run()\n```\n\n- **Get Embeddings**\n\nWe use Huggingface Accelerate to enhance efficiency:\n\n```python\nfrom deita.pipeline import Pipeline\n\nembed_pipeline = Pipeline(\"embed_pipeline\", \n                          data_path = args.data_path,   # json file with sharegpt format\n                          output_path = args.output_path,  # output path (pickle format)\n                          model_name_or_path = args.model_name_or_path,  # model name or path e.g. mistralai/Mistral-7B-v0.1\n                          max_length = args.max_length,\n                          use_flash_attention = args.use_flash_attention,  \n                          batch_size_per_device = args.batch_size_per_device,\n                          conv_template = args.conv_template,\n                          only_answer = args.only_answer,\n                          random_shuffle = args.random_shuffle,\n                          bfloat16 = True\n                          )\n\nembed_pipeline.run()\n```\n\n```bash\nCUDA_VISIBLE_DEVICES=$GPUIDX accelerate launch \\\n    --mixed_precision bf16 \\\n    --num_processes $NUMPROCESS \\\n    --num_machines 1 \\\n    examples/pipelines/embed_datasets.py \\\n    --use_flash_attention true \\\n    --data_path $DATAPATH \\\n    --output_path $OUTPUTPATH \\\n    --batch_size_per_device $BSZ\n```\n\n- **Score-first, Diversity-aware Selection**\n\n```python\nfrom deita.pipeline import Pipeline\n\nfilter_pipeline = Pipeline(\"filter_pipeline\", \n                          data_path = args.data_path,  # json file with sharegpt format\n                          other_data_path = args.other_data_path,  # embedding file path (pickle format)\n                          threshold = args.threshold,  # filter threshold default: 0.9 \n                          data_size = args.data_size,  # size of selected data\n                          chunk_size = args.chunk_size,  # used for more efficient GPU computing  default: 100000\n                          sort_key = args.sort_key,  # default: \"complexity_scores,quality_scores\"\n                          output_path = args.output_path,  # json format output path\n                          distance_metric = args.distance_metric,  # default: cosine\n                          embedding_field = args.embedding_field,  # default: embedding\n                          is_compression = args.is_compression,  # default: False\n                          device = args.device  # GPU IDX, default: 0\n                          )\n\nfilter_pipeline.run()\n```\n\nYou can refer to ```examples/pipelines``` for more details. A doc will also be coming soon.\n\n### SFT Training\nPlease refer to ```examples/train/sft.sh```\n```bash\ndeepspeed --include localhost:${DEVICES} --master_port 29501 src/deita/alignment/train.py \\\n    --model_name_or_path ${MODELPATH} \\\n    --data_path ${DATAPATH} \\\n    --output_dir ${OUTPUTPATH}/${RUNNAME} \\\n    --num_train_epochs 6 \\\n    --per_device_train_batch_size ${BSZPERDEV} \\\n    --per_device_eval_batch_size 1 \\\n    --gradient_accumulation_steps ${GRADACC} \\\n    --eval_steps 50 \\\n    --save_strategy \"no\" \\\n    --save_steps 100 \\\n    --save_total_limit 10 \\\n    --learning_rate 2e-5 \\\n    --warmup_ratio 0.1 \\\n    --lr_scheduler_type \"cosine\" \\\n    --logging_steps 1 \\\n    --do_eval False \\\n    --evaluation_strategy \"no\" \\\n    --model_max_length 2048 \\\n    --lazy_preprocess True \\\n    --conv_template \"vicuna_v1.1\" \\\n    --mask_user True \\\n    --report_to \"wandb\" \\\n    --run_name ${RUNNAME} \\\n    --bf16 True \\\n    --deepspeed src/deita/ds_configs/deepspeed_config_zero2_no_offload.json\n```\n\n### DPO Training\nPlease refer to ```examples/train/dpo.sh```\n```bash\ndeepspeed --include localhost:${DEVICES} --master_port 29502 src/deita/alignment/dpo_train.py \\\n    --model_name_or_path ${MODELPATH} \\\n    --json_path ${JSONPATH} \\\n    --data_split ${DATASPLIT} \\\n    --output_dir ${OUTPUTPATH}/${RUNNAME} \\\n    --num_train_epochs ${DPOEPOCH} \\\n    --beta 0.1 \\\n    --per_device_train_batch_size ${BSZPERDEV} \\\n    --per_device_eval_batch_size 1 \\\n    --gradient_accumulation_steps ${GRADACC} \\\n    --save_global_steps False \\\n    --eval_steps 50 \\\n    --save_strategy \"no\" \\\n    --save_steps 500 \\\n    --save_total_limit 1 \\\n    --learning_rate 5e-7 \\\n    --warmup_ratio 0.1 \\\n    --lr_scheduler_type \"linear\" \\\n    --logging_steps 1 \\\n    --do_eval False \\\n    --evaluation_strategy \"no\" \\\n    --model_max_length 2048 \\\n    --conv_template \"vicuna_v1.1\" \\\n    --report_to \"wandb\" \\\n    --run_name ${RUNNAME} \\\n    --bf16 True \\\n    --gradient_checkpointing True \\\n    --deepspeed src/deita/ds_configs/stage3_no_offloading_accelerate.json\n```\n\n### Evaluation\n- For MT-Bench, please refer to [MT-Bench](https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge)\n- For AlpacaEval, please refer to [alpaca_eval](https://github.com/tatsu-lab/alpaca_eval)\n- For Open LLM Benchmark, please refer to [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness/tree/master) and follow settings on [HuggingFaceH4/open_llm_leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)\n\n## :muscle: What's more?\n\nThis is the preview version of Deita project. We will continue to update including\n\n- [ ] Release data selection pipeline with efficient implementation\n- [ ] More automatic data selection strategies\n- [ ] CLI-Interface Supported\n- [ ] Online Demo\n\n## Citation\nIf you find the content of this project helpful, please cite our paper as follows:\n\n```\n@inproceedings{\nliu2024what,\ntitle={What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning},\nauthor={Wei Liu and Weihao Zeng and Keqing He and Yong Jiang and Junxian He},\nbooktitle={The Twelfth International Conference on Learning Representations},\nyear={2024},\nurl={https://openreview.net/forum?id=BTKAeLqLMw}\n}\n```\n\n## Acknowledgement\nFor training code, we use the code template of [fastchat](https://github.com/lm-sys/FastChat).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhkust-nlp%2Fdeita","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhkust-nlp%2Fdeita","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhkust-nlp%2Fdeita/lists"}