{"id":14801864,"url":"https://github.com/rlhf-v/rlaif-v","last_synced_at":"2025-09-15T23:32:39.241Z","repository":{"id":240613991,"uuid":"799843755","full_name":"RLHF-V/RLAIF-V","owner":"RLHF-V","description":"RLAIF-V: Aligning MLLMs through Open-Source AI Feedback for Super GPT-4V Trustworthiness","archived":false,"fork":false,"pushed_at":"2024-12-07T18:54:57.000Z","size":59120,"stargazers_count":250,"open_issues_count":5,"forks_count":10,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-12-07T19:31:43.210Z","etag":null,"topics":["chatbot","gpt-4v","llava","llava-next","minicpm-v","multimodal","rlaif-v","vision-language-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RLHF-V.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-13T07:58:37.000Z","updated_at":"2024-12-07T18:55:00.000Z","dependencies_parsed_at":"2024-11-03T06:32:09.056Z","dependency_job_id":"61f692f4-cecd-490b-882b-826e918bd996","html_url":"https://github.com/RLHF-V/RLAIF-V","commit_stats":null,"previous_names":["rlhf-v/rlaif-v"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RLHF-V%2FRLAIF-V","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RLHF-V%2FRLAIF-V/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RLHF-V%2FRLAIF-V/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RLHF-V%2FRLAIF-V/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RLHF-V","download_url":"https://codeload.github.com/RLHF-V/RLAIF-V/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233172085,"owners_count":18635927,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chatbot","gpt-4v","llava","llava-next","minicpm-v","multimodal","rlaif-v","vision-language-learning"],"created_at":"2024-09-17T20:01:00.830Z","updated_at":"2025-09-15T23:32:39.229Z","avatar_url":"https://github.com/RLHF-V.png","language":"Python","funding_links":[],"categories":["Chatbots"],"sub_categories":[],"readme":"\r\n\u003cdiv align=\"center\" style=\"font-size: 15pt\"\u003e\r\n\r\n\u003cimg src=\"examples/logo.png\" width=\"30%\" alt=\"RLAIF-V\" /\u003e\r\n\r\n**Open-Source AI Feedback Leads to Super GPT-4V Trustworthiness**\r\n\r\n\u003ca href='https://arxiv.org/abs/2405.17220'\u003e\u003cimg src='https://img.shields.io/badge/Paper-PDF-purple'\u003e\u003c/a\u003e\r\n\u003ca href='https://huggingface.co/datasets/openbmb/RLAIF-V-Dataset'\u003e\u003cimg src='https://img.shields.io/badge/Dataset-HF-Green'\u003e\u003c/a\u003e\r\n\u003ca href='https://huggingface.co/openbmb/RLAIF-V-7B'\u003e\u003cimg src='https://img.shields.io/badge/Model-7B-orange'\u003e\u003c/a\u003e\r\n\u003ca href='https://huggingface.co/openbmb/RLAIF-V-12B'\u003e\u003cimg src='https://img.shields.io/badge/Model-12B-orange'\u003e\u003c/a\u003e\r\n\r\n\u003ch4 align=\"center\"\u003e\r\n    \u003cp\u003e\r\n        \u003ca href=\"README_zh.md\"\u003e中文\u003c/a\u003e | \u003cb\u003eEnglish\u003c/b\u003e\r\n    \u003c/p\u003e\r\n\u003c/h4\u003e\r\n\r\n\u003c/div\u003e\r\n\r\n\r\n## 🎊 News \u003c!-- omit in toc --\u003e\r\n\r\n- [2025.03.01] 🎉 Our RLAIF-V is accepted by CVPR 2025! You can access the lastest version of the paper at [here](https://arxiv.org/abs/2405.17220).\r\n- [2024.11.26] 🚀 We support [LoRA](https://github.com/RLHF-V/RLAIF-V?tab=readme-ov-file#train) training now!\r\n- [2024.05.28] 📃 Our paper is accesible at [arXiv](https://arxiv.org/abs/2405.17220) now!\r\n- [2024.05.20] 🔥 Our [RLAIF-V-Dataset](https://huggingface.co/datasets/openbmb/RLAIF-V-Dataset) is used for training [MiniCPM-Llama3-V 2.5](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5), which represents the first end-side  GPT-4V level MLLM!\r\n- [2024.05.20] We open-source the code, weights ([7B](https://huggingface.co/openbmb/RLAIF-V-7B), [12B](https://huggingface.co/openbmb/RLAIF-V-12B)) and [data](https://huggingface.co/datasets/openbmb/RLAIF-V-Dataset) of RLAIF-V!\r\n\r\n\r\n## 📜 Brief Introduction \u003c!-- omit in toc --\u003e\r\n\r\nWe introduce RLAIF-V, a novel framework that aligns MLLMs in a fully open-source paradigm for super GPT-4V trustworthiness.  RLAIF-V maximally exploits the open-source feedback from two key perspectives, including high-quality feedback data and online feedback learning algorithm. Notable features of RLAIF-V include:\r\n\r\n* 💪 **Super GPT-4V Trustworthiness via Open-source Feedback**. By learning from open-source AI feedback, RLAIF-V 12B achieves super GPT-4V trustworthiness in both generative and discriminative tasks.\r\n\r\n\u003ctable align=\"center\"\u003e\r\n    \u003cp align=\"center\"\u003e\r\n      \u003cimg src=\"examples/introduction1.png\" width=\"80%\" alt=\"introduction1\" /\u003e\r\n    \u003c/p\u003e\r\n\u003c/table\u003e\r\n\r\n\r\n\r\n* 🤝 **High-quality Generalizable Feedback Data (Figure 4)**. The feedback data used by RLAIF-V **effectively reduce the hallucination of different MLLMs**.\r\n\r\n\u003ctable align=\"center\"\u003e\r\n    \u003cp align=\"center\"\u003e\r\n      \u003cimg src=\"examples/introduction2.jpg\" width=\"80%\" alt=\"introduction2\" /\u003e\r\n    \u003c/p\u003e\r\n\u003c/table\u003e\r\n\r\n\r\n* 🚀 **Inference-time Scaling by RLAIF-V reward (Figure 5)**. RLAIF-V reward consistently improves the trustworthiness performance of different models when scaling up inference budgets.\r\n\r\n\r\n## 📌Contents \u003c!-- omit in toc --\u003e\r\n\r\n- [Dataset](#dataset)\r\n- [Install](#install)\r\n- [Model Weights](#model-weights)\r\n- [Inference](#inference)\r\n- [Data Generation](#data-generation)\r\n- [Train](#train)\r\n- [Evaluation](#evaluation)\r\n  - [Object HalBench](#object-halbench)\r\n  - [MMHal Bench](#mmhal-bench)\r\n  - [RefoMB](#refomb)\r\n- [Citation](#citation)\r\n\r\n## Dataset\r\n\r\nWe present the [RLAIF-V Dataset](https://huggingface.co/datasets/openbmb/RLAIF-V-Dataset), which is an AI generated preference dataset covering diverse range of tasks and domains. This open-source multimodal preference datasets contains **83,132 high-quality comparison pairs**. The dataset contains the generated preference pairs in each training iteration of different models, including LLaVA 1.5 7B, OmniLMM 12B and MiniCPM-V.\r\n\r\n## Install\r\n\r\n1. Clone this repository and navigate to RLAIF-V folder\r\n```bash\r\ngit clone https://github.com/RLHF-V/RLAIF-V.git\r\ncd RLAIF-V\r\n```\r\n\r\n2. Install package\r\n```bash\r\nconda create -n rlaifv python=3.10 -y\r\nconda activate rlaifv\r\npip install -e .\r\n```\r\n3. Install required spaCy model\r\n```bash\r\nwget https://github.com/explosion/spacy-models/releases/download/en_core_web_trf-3.7.3/en_core_web_trf-3.7.3.tar.gz\r\npip install en_core_web_trf-3.7.3.tar.gz\r\n```\r\n\r\n\r\n## Model Weights\r\n\r\n\r\n| Model           | Description    | Download                                                    |\r\n|-----------------|--------------------|:-:|\r\n| RLAIF-V 7B  | The most trustworthy variant on LLaVA 1.5 | [🤗](https://huggingface.co/openBMB/RLAIF-V-7B) |\r\n| RLAIF-V 12B | Based on OmniLMM-12B, achieving super GPT-4V trustworthiness. | [🤗](https://huggingface.co/openBMB/RLAIF-V-12B)           |\r\n\r\n## Inference\r\n\r\nWe provide a simple example to show how to use RLAIF-V.\r\n\r\n\r\n```python\r\n\r\nfrom chat import RLAIFVChat, img2base64\r\n\r\nchat_model = RLAIFVChat('openBMB/RLAIF-V-7B')  # or 'openBMB/RLAIF-V-12B'\r\nimage_path=\"./examples/test.jpeg\"\r\nmsgs = \"Describe in detail the people in the picture.\"\r\ninputs = {\"image\": image_path, \"question\": msgs}\r\nanswer = chat_model.chat(inputs)\r\nprint(answer)\r\n\r\n```\r\n\r\n\r\nYou can also run this example by executing the following script:\r\n\r\n```bash\r\npython chat.py\r\n```\r\n\r\n\u003cdetails\u003e\r\n  \u003csummary\u003e\r\n    \u003cb\u003eInputs and expected outputs of the example\u003c/b\u003e\r\n  \u003c/summary\u003e\r\n\r\n\r\n\r\n\u003cdiv align=\"center\"\u003e\r\n\u003cimg src=\"examples/test.jpeg\" width=\"500px\"\u003e\r\n\u003c/div\u003e\r\n\r\n**Question:**\r\n\r\nWhy did the car in the picture stop?\r\n\r\n**Expected outputs:**\r\n\r\nIn the picture, a car stopped on the road due to the presence of a sheep on the roadway. The car likely stopped to allow the sheep to safely move out of the way or avoid any potential accidents with the animal. This situation highlights the importance of being cautious and attentive while driving, especially in areas where animals may roam near roads.\r\n\r\n\u003c/details\u003e\r\n\r\n\r\n## Data Generation\r\n1. Environment Setup\r\n\r\nWe provide the OmniLMM 12B model and the MiniCPM-Llama3-V 2.5 model for feedback generation. If you wish to use the MiniCPM-Llama3-V 2.5 for giving feedback, please configure its inference environment according to the instructions in the [MiniCPM-V GitHub repository](https://github.com/OpenBMB/MiniCPM-V).\r\n\r\nPlease download our fine-tuned Llama3 8B models: [split model](https://thunlp.oss-cn-qingdao.aliyuncs.com/rlaifv_llama3_split_model.tar.gz) and [question transformation model](https://thunlp.oss-cn-qingdao.aliyuncs.com/rlaifv_llama3_changeq_model.tar.gz), and store them in the `./models/llama3_split` folder and the `./models/llama3_changeq` folder respectively.\r\n\r\n2. OmniLMM 12B Model Feedback\r\n\r\nThe following script demonstrates using the LLaVA-v1.5-7b model to generate candidate answers and the OmniLMM 12B model to provide feedback.\r\n\r\n```bash\r\nmkdir ./results\r\nbash ./script/data_gen/run_data_pipeline_llava15_omni.sh\r\n```\r\n\r\n3. MiniCPM-Llama3-V 2.5 Model Feedback\r\n\r\nThe following script demonstrates using the LLaVA-v1.5-7b model to generate candidate answers and the MiniCPM-Llama3-V 2.5 model to provide feedback. First, replace `minicpmv_python` in `./script/data_gen/run_data_pipeline_llava15_minicpmv.sh` with the Python path of the MiniCPM-V environment you created.\r\n\r\n```bash\r\nmkdir ./results\r\nbash ./script/data_gen/run_data_pipeline_llava15_minicpmv.sh\r\n```\r\n\r\n\r\n## Train\r\n\r\n1. Prepare data (Optional)\r\n\r\nIf you can access huggingface dataset, you can skip this step, we will automatically download the [RLAIF-V Dataset](https://huggingface.co/datasets/openbmb/RLAIF-V-Dataset).\r\n\r\nIf you already downloaded the dataset, you can replace 'openbmb/RLAIF-V-Dataset' to your dataset path [here](muffin/data/datasets.py#L38) at Line 38.\r\n\r\n2. Training\r\n\r\nHere, we provide a training script to train the model in **1 iteration**. The `max_step` parameter should be adjusted according to the amount of your data.\r\n\r\n- **Fully Fine-tuning**\r\n  \r\nRun the following command to start fully fine-tuning.\r\n\r\n```bash\r\nbash ./script/train/llava15_train.sh\r\n```\r\n\r\n- **LoRA**\r\n\r\nRun the following command to start lora training.\r\n\r\n```bash\r\npip install peft \r\nbash ./script/train/llava15_train_lora.sh\r\n```\r\n3. Iterative alignment\r\n\r\nTo reproduce the iterative training process in the paper, you need to do the following steps for 4 times:\r\n- **S1. Data generation.**\r\n\r\n  Follow the instructions in [data generation](https://github.com/RLHF-V/RLAIF-V?tab=readme-ov-file#data-generation) to generate preference pairs for the base model. Convert the generated jsonl file to huggingface parquet.\r\n- **S2. Change training config.**\r\n\r\n  In dataset code, replace `'openbmb/RLAIF-V-Dataset'` [here](muffin/data/datasets.py#L38) to your data path.\r\n\r\n  In [training script](script/train/llava15_train.sh), replace `--data_dir` with a new directory, replace `--model_name_or_path` with the base model path, set `--max_step` to the number of steps for 4 epoch, set `--save_steps` to the number of steps for 1/4 epoch.\r\n- **S3. Do DPO training.**\r\n\r\n  Run the training script to train the base model.\r\n- **S4. Choose base model for next iteration.**\r\n\r\n  Evaluate each checkpoint on Object HalBench and MMHal Bench, choose the best-performed checkpoint as the base model in the next iteration.\r\n\r\n## Evaluation\r\n\r\n### Object HalBench\r\n\r\n1. Prepare COCO2014 annotations\r\n\r\nThe evaluation of Object HalBench relies on the caption and segmentation annotations from the COCO2014 dataset. Please first download the COCO2014 dataset from the COCO dataset's official website.\r\n\r\n```bash\r\nmkdir coco2014\r\ncd coco2014\r\n\r\nwget http://images.cocodataset.org/annotations/annotations_trainval2014.zip\r\n\r\nunzip annotations_trainval2014.zip\r\n```\r\n\r\n2. Inference, evaluation, and summarization\r\n\r\nPlease replace `{YOUR_OPENAI_API_KEY}` with a valid OpenAI api-key.\r\n\r\n**Note: The evaluation is based on `gpt-3.5-turbo-0613`.**\r\n\r\n```bash\r\n# cd RLAIF-V\r\n\r\nbash ./script/eval/eval_rlaifv_objhal.sh ./RLAIF-V_weight ./results/RLAIF-V ./coco2014/annotations {YOUR_OPENAI_API_KEY}\r\n```\r\n\r\n\r\n### MMHal Bench\r\n\r\n1. Prepare MMHal Data\r\n\r\nPlease download the MMHal evaluation data [here](https://drive.google.com/file/d/1mQyAbeGgRyiVV6qjVkUI1uY_g9E-bDTH/view?usp=sharing), and save the file in `eval/data`.\r\n\r\n2. Run the following script to generate for MMHal Bench:\r\n\r\n**Note: The evaluation is based on `gpt-4-1106-preview`.**\r\n\r\n```bash\r\n# cd RLAIF-V\r\n\r\nbash ./script/eval/eval_rlaifv_mmhal.sh ./RLAIF-V_weight ./results/RLAIF-V {YOUR_OPENAI_API_KEY}\r\n```\r\n\r\n### RefoMB\r\n\r\n1. Preparation\r\n\r\nTo use GPT-4 evaluation, please first run `pip install openai==0.28` to install openai package. Next, change the `openai.base` and `openai.api_key` in `eval/gpt4.py` into your own setting.\r\n\r\nEvaluation data for dev set can be found at `eval/data/RefoMB_dev.jsonl`. You need to download each image from the `image_url` key in each line.\r\n\r\n2. Evaluation for overall score\r\n\r\nSave your model answer in `answer` key of the input data file `eval/data/RefoMB_dev.jsonl`, for example:\r\n\r\n```\r\n{\r\n    \"image_url\": \"https://thunlp.oss-cn-qingdao.aliyuncs.com/multimodal_openmme_test_20240319__20.jpg\",\r\n    \"question\": \"What is the background of the image?\",\r\n    \"type\": \"Coarse Perception\",\r\n    \"split\": \"dev\",\r\n    \"answer\": \"The background of the image features trees, suggesting that the scene takes place outdoors.\",\r\n    \"gt_description\": \"......\"\r\n}\r\n```\r\n\r\nRun the following script to evaluate your model result:\r\n\r\n```\r\nsave_dir=\"YOUR SAVING DIR\"\r\nmodel_ans_path=\"YOUR MODEL ANSWER PATH\"\r\nmodel_name=\"YOUR MODEL NAME\"\r\n\r\nbash ./script/eval/run_refobm_overall.sh $save_dir $model_ans_path $model_name\r\n```\r\n\r\n3. Evaluation for hallucination score\r\n\r\nAfter evaluating the overall score, an evaluation result file will be created with name `A-GPT-4V_B-${model_name}.json`. Using this evaluation result file to calculate the hallucination score as follows:\r\n\r\n```\r\neval_result=\"EVAL RESULT FILE PATH, e.g. 'A-GPT-4V_B-${model_name}'\"\r\n# Do not include \".json\" in your file path!\r\n\r\nbash ./script/eval/run_refomb_hall.sh $eval_result\r\n```\r\n\r\n4. **Note:** For better stability, we recommend you to evaluate more than **3 times** and use the **average score** as the final model score.\r\n\r\n\r\n## Licenses \u003c!-- omit in toc --\u003e\r\n\r\n\r\n[![Code License](https://img.shields.io/badge/Code%20License-Apache_2.0-green.svg)](https://github.com/tatsu-lab/stanford_alpaca/blob/main/LICENSE)\r\n[![Data License](https://img.shields.io/badge/Data%20License-CC%20By%20NC%204.0-red.svg)](https://github.com/tatsu-lab/stanford_alpaca/blob/main/DATA_LICENSE)\r\n\r\n**Usage and License Notices**: The data, code, and checkpoint are intended and licensed for research use only. They are also restricted to uses that follow the license agreement of LLaMA, Vicuna, and Chat GPT. The dataset is CC BY NC 4.0 (allowing only non-commercial use) and models trained using the dataset should not be used outside of research purposes.\r\n\r\n\r\n\r\n## Acknowledgement \u003c!-- omit in toc --\u003e\r\n\r\n- [RLHF-V](https://github.com/RLHF-V/RLHF-V): The codebase we built upon.\r\n- [LLaVA](https://github.com/haotian-liu/LLaVA): The instruction model and labeler model of RLAIF-V-7B.\r\n- [MiniCPM-V](https://github.com/OpenBMB/MiniCPM-V): The instruction model and labeler model of RLAIF-V-12B.\r\n- [MiniCPM-o](https://github.com/OpenBMB/MiniCPM-o): The end-to-end omni-modal model using RLAIF-V method for alignment.\r\n\r\n## Citation\r\n\r\nIf you find our model/code/data/paper helpful, please consider cite our papers 📝 and star us ⭐️！\r\n\r\n```bibtex\r\n@article{yu2023rlhf,\r\n  title={Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback},\r\n  author={Yu, Tianyu and Yao, Yuan and Zhang, Haoye and He, Taiwen and Han, Yifeng and Cui, Ganqu and Hu, Jinyi and Liu, Zhiyuan and Zheng, Hai-Tao and Sun, Maosong and others},\r\n  journal={arXiv preprint arXiv:2312.00849},\r\n  year={2023}\r\n}\r\n\r\n@article{yu2024rlaifv,\r\n  title={RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthiness},\r\n  author={Tianyu Yu and Haoye Zhang and Qiming Li and Qixin Xu and Yuan Yao and Da Chen and Xiaoman Lu and Ganqu Cui and Yunkai Dang and Taiwen He and Xiaocheng Feng and Jun Song and Bo Zheng and Zhiyuan Liu and Tat-Seng Chua and Maosong Sun},\r\n  journal={arXiv preprint arXiv:2405.17220},\r\n  year={2024},\r\n}\r\n```\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frlhf-v%2Frlaif-v","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frlhf-v%2Frlaif-v","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frlhf-v%2Frlaif-v/lists"}