https://github.com/sail-sg/dice
Official implementation of Bootstrapping Language Models via DPO Implicit Rewards
https://github.com/sail-sg/dice
alignment large-language-models preference-learning rlhf
Last synced: 11 months ago
JSON representation
Official implementation of Bootstrapping Language Models via DPO Implicit Rewards
- Host: GitHub
- URL: https://github.com/sail-sg/dice
- Owner: sail-sg
- License: mit
- Created: 2024-06-12T09:44:09.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2025-04-15T01:48:27.000Z (about 1 year ago)
- Last Synced: 2025-06-09T15:53:00.061Z (about 1 year ago)
- Topics: alignment, large-language-models, preference-learning, rlhf
- Language: Python
- Homepage: https://arxiv.org/abs/2406.09760
- Size: 18.4 MB
- Stars: 44
- Watchers: 3
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Bootstrapping with DPO Implicit Rewards (DICE)
[](https://huggingface.co/collections/sail/dice-6684de998e62fe07709d67eb)
[](https://arxiv.org/abs/2406.09760)
[](https://github.com/sail-sg/dice/blob/main/LICENSE)
This repository contains the implementation of our paper Bootstrapping Language Models via DPO Implicit Rewards. We show that the implicit reward model from the prior DPO training can be utilized to bootstrap and further align LLMs.
## Quick links
- [Bootstrapping with DPO Implicit Rewards (DICE)](#bootstrapping-with-dpo-implicit-rewards-dice)
- [Quick links](#quick-links)
- [Base Models and Released Models](#base-models-and-released-models)
- [Setup](#setup)
- [Install dependencies](#install-dependencies)
- [Setup the bash script](#setup-the-bash-script)
- [Training scripts](#training-scripts)
- [Acknowledgement](#acknowledgement)
- [Citation](#citation)
## Base Models and Released Models
| **Model** | **AE2 LC** | **AE2 WR** |
|----------------------------|:----------:|:----------:|
| 🤗[Llama-3-Base-8B-SFT-DPO](https://huggingface.co/princeton-nlp/Llama-3-Base-8B-SFT-DPO) | 18.20 | 15.50 |
| 🤗[Llama-3-Base-8B-DICE Iter1](https://huggingface.co/sail/Llama-3-Base-8B-DICE-Iter1) | 25.08 | 25.77 |
| 🤗[Llama-3-Base-8B-DICE Iter2](https://huggingface.co/sail/Llama-3-Base-8B-DICE-Iter2) | 27.55 | 30.99 |
| 🤗[Zephyr-7b-beta](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta) | 12.69 | 10.71 |
| 🤗[Zephyr-7B-DICE Iter1](https://huggingface.co/sail/Zephyr-7B-DICE-Iter1) | 19.03 | 17.67 |
| 🤗[Zephyr-7B-DICE Iter2](https://huggingface.co/sail/Zephyr-7B-DICE-Iter2) | 20.71 | 20.16 |
Please refer to [pipeline.sh#1.1_response_generation](https://github.com/sail-sg/dice/blob/21abbe8c44ad2d608dbcf14551c209064ce66540/scripts/run_dice/pipeline.sh#L105) on instructions for batch inference with the appropriate chat template.
## Setup
### Install dependencies
Please install dependencies using the following command:
```bash
git clone https://github.com/sail-sg/dice.git
conda create -n dice python=3.10
conda activate dice
cd dice/llama-factory
pip install -e .[deepspeed,metrics,bitsandbytes]
cd ..
pip install -e .
pip install -r requirements.txt
# optional to install flash attention
pip install flash-attn --no-build-isolation
```
### Setup the bash script
Provide the local path to this repo to `DICE_DIR` in two files:
- `scripts/run_dice/iter.sh`
- `scripts/run_dice/pipeline.sh`
E.g. `DICE_DIR="/home/username/dice"`
## Training scripts
We provide sample training scripts for both Llama3 and Zephyr settings. It is recommended to run the script with `8x A100 GPUs`. For other hardware environments, you might need to adjust the script.
- Llama3
```bash
bash scripts/run_dice/iter.sh llama3
```
- Zephyr
```bash
bash scripts/run_dice/iter.sh zephyr
```
## Acknowledgement
This repo is built on [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory). Thanks for the amazing work!
## Citation
Please consider citing our paper if you find the repo helpful in your work:
```bibtex
@inproceedings{chen2025bootstrapping,
title={Bootstrapping Language Models with DPO Implicit Rewards},
author={Chen, Changyu and Liu, Zichen and Du, Chao and Pang, Tianyu and Liu, Qian and Sinha, Arunesh and Varakantham, Pradeep and Lin, Min},
booktitle={International Conference on Learning Representations (ICLR)},
year={2025}
}
```