{"id":26123352,"url":"https://github.com/gair-nlp/realign","last_synced_at":"2025-04-13T14:34:40.440Z","repository":{"id":217888486,"uuid":"649629823","full_name":"GAIR-NLP/ReAlign","owner":"GAIR-NLP","description":"Reformatted Alignment","archived":false,"fork":false,"pushed_at":"2024-09-23T15:44:00.000Z","size":39390,"stargazers_count":115,"open_issues_count":0,"forks_count":7,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-27T05:34:30.937Z","etag":null,"topics":["alignment","generative-ai","large-language-models","llms","natural-language-processing","nlp"],"latest_commit_sha":null,"homepage":"https://gair-nlp.github.io/ReAlign/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GAIR-NLP.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-06-05T09:45:56.000Z","updated_at":"2025-03-19T12:19:56.000Z","dependencies_parsed_at":"2024-02-23T15:42:38.267Z","dependency_job_id":"f043bbd0-9804-4028-bf0a-dc6ffd1ed139","html_url":"https://github.com/GAIR-NLP/ReAlign","commit_stats":null,"previous_names":["gair-nlp/realign"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GAIR-NLP%2FReAlign","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GAIR-NLP%2FReAlign/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GAIR-NLP%2FReAlign/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GAIR-NLP%2FReAlign/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GAIR-NLP","download_url":"https://codeload.github.com/GAIR-NLP/ReAlign/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248728696,"owners_count":21152277,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["alignment","generative-ai","large-language-models","llms","natural-language-processing","nlp"],"created_at":"2025-03-10T15:53:00.061Z","updated_at":"2025-04-13T14:34:40.405Z","avatar_url":"https://github.com/GAIR-NLP.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Reformatted Alignment\nThis is the official repository for [**Reformatted Alignment**](https://arxiv.org/abs/2402.12219).\n\n[Run-Ze Fan](https://scholar.google.com/citations?user=mhot7AUAAAAJ\u0026hl=en), [Xuefeng Li](https://github.com/hongtangshui), [Haoyang Zou](https://openreview.net/profile?id=~Haoyang_Zou1), [Junlong Li](https://lockon-n.github.io/), [Shwai He](https://shwai-he.github.io/), [Ethan Chern](https://ethanc111.github.io/), [Jiewen Hu](https://www.linkedin.com/in/jiewen-hu/), [Pengfei Liu](http://pfliu.com/)\n\n## News\n- **Sep 2024**: Our paper has been accepted by EMNLP 2024 Findings! 🎉\n- **Feb 2024**: We release the preprint paper on Arxiv, ReAlign data, and other useful resources in developing them (tasks description, hand-written format, tasks classifier, training data, and NQ dataset for factuality evaluation).\n\n## Table of contents\n\n- [Introduction](#Introduction)\n- [Quick Start](#quick-start)\n  - [Setup](#setup)\n  - [Pipeline](#pipeline)\n- [ReAlign Dataset](#realign-dataset)\n- [Other Resources](#other-resources)\n  - [Tasks Description and Formats](#tasks-description-and-formats)\n  - [The Data for Task Classifier](#the-data-for-task-classifier)\n  - [Factuality Evaluation: NQ Dataset](#factuality-evaluation)\n- [Citation](#citation)\n- [Acknowledgements](#acknowledgements)\n\n## Introduction\nWe explores elevating the quality of existing instruction data to better align with human values, introducing a simple and effective approach named **ReAlign** (**Re**formatted **Align**ment), which *reformats* the responses of instruction data into a format that better aligns with pre-established criteria and the collated evidence.\nThis approach minimizes human annotation, hallucination, and the difficulty in scaling, remaining orthogonal to existing alignment techniques.\nExperimentally, ReAlign significantly boosts the general alignment ability, math reasoning, factuality, and readability of the LLMs.\n\nEncouragingly, *without* introducing any additional data or advanced training techniques, and merely by reformatting the response, LLaMA-2-13B's mathematical reasoning ability on GSM8K can be improved **from 46.77% to 56.63%** in accuracy.\nAdditionally, a mere 5% of ReAlign data yields a 67% boost in general alignment ability measured by the Alpaca dataset. \nThis work highlights the need for further research into the *science* and *mechanistic interpretability* of LLMs.\n\nThe underlying *philosophy* of ReAlign is to re-coordinate the roles of humans and LLMs in the alignment process, leveraging their complementary strengths -- humans articulate their preferences, and LLMs, in turn, reconstruct instructions based on their generative power (e.g., instruction-following ability), without directly using distilled LLM knowledge.\nThrough this collaborative synergy, we expect the generated instruction data to be not only more contextually precise but also more closely aligned with human preferences.\n\n\n\u003cdiv align=center\u003e\u003cimg src=\"./figs/Math_Results.jpg\" style=\"zoom: 25%;\" /\u003e\u003c/div\u003e\n\u003ccenter\u003eThe accuracy of the GSM8K test set for LLaMA-2-13B and Mistral-7B models fine-tuned on the training set of GSM8K and MATH with and without ReAlign. (a): Training and testing on GSM8K. (b): Training on MATH and testing on GSM8K (Out-of-Distribution Setting).\u003c/center\u003e\n\n\n\u003cdiv align=center\u003e\u003cimg src=\"./figs/overall_figs.jpg\" style=\"zoom: 25%;\" /\u003e\u003c/div\u003e\n\u003ccenter\u003eAn overview of our \u003cstrong\u003eReAlign\u003c/strong\u003e including three steps. KILT denotes Knowledge Intensive Language Tasks.\u003c/center\u003e\n\nThe ReAlign process unfolds in three main steps. \n\nThe first step involves **criteria definition**, where humans define their preferences (e.g., the preferred format of responses) in various scenarios in the form of natural language.\nIn this paper, we meticulously define criteria for 46 distinct scenarios. \n\nThe second step, **retrieval augmentation**, broadens the knowledge base for knowledge-intensive tasks like open-domain QA and fact verification. This is achieved by incorporating additional information, thereby improving the factuality and informativeness of responses. \n\nThe final step, **reformatting**, aims to re-align the responses with the pre-established criteria and the collated evidence, guaranteeing outputs that are both structured and substantiated.\n\n\u003cdiv align=center\u003e\u003cimg src=\"./figs/intro_graph.jpg\" style=\"zoom: 25%;\" width=\"60%\" height=\"auto\"/\u003e\u003c/div\u003e\n\u003ccenter\u003e\u003cstrong\u003eReAlign\u003c/strong\u003e realigns the original response with the pre-defined criteria to be a better format.\u003c/center\u003e\n\n\u003cdiv align=center\u003e\u003cimg src=\"./figs/model_example.jpg\" style=\"zoom: 25%;\" /\u003e\u003c/div\u003e\n\u003ccenter\u003eAn example of the response from the original model and the response from the ReAlign Model\u003c/center\u003e\n\n## Quick Start\n\n### Setup\nWe use `python 3.10` in this project. You are encouraged to create a virtual environment through `conda`.\n\nThen, we have to install all the libraries listed in `requirements.txt`. Note that you may choose an appropriate version of `torch` according to your CUDA version (we write `torch\u003e=2.0.1+cu118` in this file).\n\n```bash\npip install -r requirements.txt\n```\n\n### Pipeline\n* get your OpenAI API key from [here](https://beta.openai.com/). This is used for reformatting.\n* get your Serper API key from [here](https://serper.dev/). This is only used for retrieval with Google Search. \n\n\n#### Step 1: Task Classification\nDownload the task classifier from huggingface hub:\n\n| Model Name      | HF Checkpoints                                                  | Size    | License                                                      |\n|-----------------|-----------------------------------------------------------------| ------- | ------------------------------------------------------------ |\n| Task Classifier | [🤗 GAIR/ReAlign-Task-Classifier](https://huggingface.co/GAIR/ReAlign-Task-Classifier) | **13B** | [Llama 2](https://ai.meta.com/resources/models-and-libraries/llama-downloads/) |\n\nThen, by using the following prompt, the task classifier can identify which task a query belongs to:\n```python\nPROMPT_INPUT_FOR_TASK_CLS: str = '''\nYou will receive a user's query. Additionally, you are given some pre-defined tasks below: \n\n[Existing tasks start]\nquestion_generation\nstory_generation\npoem_generation\nemail_generation\ndata_generation\nadvice_giving\nrecommendations\nhow_to_generation\nplanning\ninstructional_rewriting\nlanguage_polishing\nparaphrasing\ntext_correction\ncode_correction\ncode_simplification\ninformation_extraction\nkeywords_extraction\ntable_extraction\ntitle_generation\ntext_summarization\nnote_summarization\nexplain_code\nexplain_answer\ntext_to_text_translation\ntext_to_code_translation\ncode_to_code_translation\ncode_to_text_translation\nopen_qa\nclosed_qa\nfill_in_the_blank\nfact_verification\nmath_puzzles\nlanguage_learning_questions\nnatural_language_learning_tutor\nexam_problem_solving_tutor\nml_ai_language_model_tutor\ngeneral_classification\nordering\nsentiment_analysis\ncode_language_classification\nlanguage_classification\ntopic_classification\nvalue_judgement\nrejecting\nroleplay\ndefault\n[Existing tasks end]\n\nYou objective is to choose the most appropriate task that can reflect the high-level intention of this query. You should first clearly give out your choice. Your choice should exactly match one of the task names provided above, without any modification. Do not include the task description in your choice.\n\nYour output should be just the task name.\n\nUser's query is below:\n[User's query start]\n{input}\n[User's query end]\n\nTask name:\n\n'''\n```\nHere is an example:\n```python\nfrom vllm import LLM, SamplingParams\nimport torch\n\nnum_gpus = torch.cuda.device_count()\nmodel_name_or_dir = \"GAIR/ReAlign-Task-Classifier\" # or the local directory to store the downloaded model\nllm = LLM(model=model_name_or_dir, tensor_parallel_size=num_gpus)\n\nquery = \"Give three tips for staying healthy.\"\ninput_ = PROMPT_INPUT_FOR_TASK_CLS.format(input=query)\n\nsampling_params = SamplingParams(temperature=0.0, top_p=1.0, max_tokens=50)\noutputs = llm.generate(input_, sampling_params)\ntask = output[0].outputs[0].text\n\nprint(task) # should be `advice_giving`.\n# If the classified result is not in task list, set it as `default`.\n```\n\n\n#### Step 2: Prepare your dataset\nConvert your dataset into the following format with json type, same as the ReAlign datasets.\n\nHere is an example:\n```python\n[\n    {\n        \"id\": 0,\n        \"items\": [\n            {\n                # question\n                \"from\": \"human\",\n                \"value\": \"Give three tips for staying healthy.\",\n                \"category\": \"advice_giving\"\n            },\n            {\n                # response\n                \"from\": \"gpt\",\n                \"value\": \"1.Eat a balanced diet and make sure to include plenty of fruits and vegetables. \\n2. Exercise regularly to keep your body active and strong. \\n3. Get enough sleep and maintain a consistent sleep schedule.\"\n            }\n        ]\n    }\n]\n```\n\n#### Step 3: Retrieval with Google Search\nSet your Serper API key: \n```python\nexport SERPER_API_KEY=...\n```\nRun the following script:\n```python\npython retrieval.py \\\n    --input_data_path dataset.json \\\n    --output_path dataset_retrieval.json \\\n    --batch_size 10\n```\nThe output file:\n\n`dataset_retrieval.json`is added the original retrieval results.\n\n`dataset_retrieval_clean_evidence.json`is added the cleaned retrieval results.\nThis is used for ReAlign.\n\n#### Step 4: Reformat\nSet your OpenAI API key: \n```python\nexport OPENAI_API_KEY=...\n```\n\nRun the following script:\n```python\npython reformat.py \\\n    --input_data_path dataset_retrieval_clean_evidence.json \\\n    --output_directory reformat_results \\\n    --tokenizer_path meta-llama/Llama-2-7b-chat-hf \\ # or the local directory to store the downloaded tokenizer\n    --dataset_batch_id 0 \\ # the first file (it's in 0 - 9) of ten files\n    --dataset_batch_num 10 \\ # the total number of the file\n    --openai_key \u003cOPENAI_API_KEY\u003e \\\n    --top_k 2 \\ # output 2 reformatted response for each response\n    --model gpt-3.5-turbo-1106 \\\n    --temperature 0.3 \\\n    --top_p 1 \\\n    --target_length 4096\n```\n\nNote that we are using process parallel to speedup, which means that we are going to run `dataset_batch_num` processes at the same time for reformatting, and each process will need to specify the `dataset_batch_id` manually.\n\nFor example:\n\nIf you set `dataset_batch_num` as 10, it means the datasets will be split into 10 subdataset (10x acceleration). You should run the script 10 times at the same time, each time specifying `dataset_batch_id` as 0 through 9.\n\nThen, you can get `dataset_batch_num` files in the directory `output_directory`.\n\nRun the following script to merge these files into one final datasets:\n```python\npython parallel_data_merge.py \\\n    --input_data_path dataset_retrieval_clean_evidence.json \\ # the \u003cinput_data_path\u003e in reformat script\n    --output_directory reformat_results \\ # the \u003coutput_directory\u003e in reformat script\n    --final_output_path dataset_reformat.json\n```\nFinally, you can get the final reformatted datasets.\n\n#### Step 5: Post Filtering\nYou can combine the filtering rules in `rewrite_data_selection.py` or customize the filtering rules.\n\nRun the following script to filter the reformatted dataset:\n```python\npython rewrite_data_selection.py \\\n    --input_original_data_path dataset_retrieval_clean_evidence.json \\ # the dataset path before reformatting\n    --input_rewrite_data_path dataset_reformat.json \\ # the reformatted dataset path\n    --output_path realign_dataset.json # the final dataset path after filtering\n```\n\nNow, you can get the final realign dataset `realign_dataset.json`.\n\n## ReAlign Dataset\nWe reformat five datasets based Open-Platypus, Alpaca, No Robots, GSM8K, and MATH:\n\nReAlign Open-Platypus: `datasets/realign_OpenPlatypus.json`\n\nReAlign Alpaca: `datasets/realign_alpaca.json`\n\nReAlign No Robots: `datasets/realign_no_robots.json`\n\nReAlign GSM8K: `datasets/realign_gsm8k.json`\n\nReAlign MATH: `datasets/realign_math.json`\n\nThe datasets also can be loaded on 🤗Hugging Face:\n\n| Dataset Name          | Hugging Face Link | Size |\n|-----------------------|-----------------------------------------------------------|------|\n| ReAlign Open-Platypus | [🤗 GAIR/ReAlign-Open-Platypus](https://huggingface.co/datasets/GAIR/ReAlign-Open-Platypus) | 25K  |\n| ReAlign Alpaca        | [🤗 GAIR/ReAlign-Alpaca](https://huggingface.co/datasets/GAIR/ReAlign-Alpaca) | 52K  |\n| ReAlign No Robots     | [🤗 GAIR/ReAlign-No-Robots](https://huggingface.co/datasets/GAIR/ReAlign-No-Robots) | 10K  |\n| ReAlign GSM8K         | [🤗 GAIR/ReAlign-GSM8K](https://huggingface.co/datasets/GAIR/ReAlign-GSM8K) | 7.4K |\n| ReAlign MATH          | [🤗 GAIR/ReAlign-MATH](https://huggingface.co/datasets/GAIR/ReAlign-MATH) | 6.5K |\n\n## Other Resources\n\n### Tasks Description and Formats\nThe tasks description and predefined formats can be found in `code/constant.py`.\n\n### The Data for Task Classifier\nThe training data for the task classifier is in `datasets/classification/task_classifier_train_dataset.json`.\n\nThe test data is in `datasets/classification/task_classifier_test_dataset.json`.\n\nThe format is as follows:\n```python\n{\n        \"instruction\": \"Create a story about a dog that finds a magical portal.\",\n        \"category\": \"story_generation\"\n}\n```\n\n### Factuality Evaluation\n\nWe randomly sample 100 cases from NQ Dataset for factuality evaluation, which can be found in `datasets/nq`.\n\nThe ground truth is in `datasets/nq/nq_factuality_100.json`.\n\nThe format is as follows:\n```python\n{\n        \"items\": [\n            {\n                \"from\": \"human\",\n                \"value\": \"when did the democratic party change its name?\"\n            },\n            {\n                \"from\": \"gpt\",\n                \"value\": \"the 1830s\"\n            }\n        ],\n        \"id\": 0\n}\n```\n\n## Citation\nPlease cite the paper if the resource in this repo or the paper is helpful to you.\n```\n@article{fan2024reformatted,\n      title={Reformatted Alignment}, \n      author={Fan, Run-Ze and Li, Xuefeng and Zou, Haoyang and Li, Junlong and He, Shwai and Chern, Ethan and Hu, Jiewen and Liu, Pengfei},\n      year={2024},\n      journal={arXiv preprint arXiv:2402.12219},\n      url={https://arxiv.org/abs/2402.12219}\n}\n```\n\n\n## Acknowledgements\nWe thank the GAIR members for reviewing our paper and giving valuable feedback. We appreciate the authors in [OpenChat](https://github.com/imoneoi/openchat) for providing the training codebase and the helpfulness.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgair-nlp%2Frealign","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgair-nlp%2Frealign","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgair-nlp%2Frealign/lists"}