https://github.com/zwhong714/Hybrid-Policy-Distillation
https://github.com/zwhong714/Hybrid-Policy-Distillation
Last synced: about 1 month ago
JSON representation
- Host: GitHub
- URL: https://github.com/zwhong714/Hybrid-Policy-Distillation
- Owner: zwhong714
- Created: 2026-03-21T07:12:33.000Z (4 months ago)
- Default Branch: main
- Last Pushed: 2026-04-23T04:54:05.000Z (3 months ago)
- Last Synced: 2026-04-23T06:25:49.417Z (3 months ago)
- Language: Python
- Size: 6.71 MB
- Stars: 1
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesomeopd - Hybrid-Policy-Distillation - Policy-Distillation?style=for-the-badge&logo=github&logoColor=white&labelColor=181717&color=ffd700" alt="Stars"> | 2026.04 | zwhong714 | [arXiv 2604.20244](https://arxiv.org/abs/2604.20244) | HPD — Hybrid Policy Distillation; LlamaFactory + verl backends | (🔬 OPD with Larger External Teachers — White-Box)
README
Hybrid Policy Distillation
[
Paper](https://arxiv.org/pdf/2604.20244) |
[
Code](https://github.com/zwhong714/Hybrid-Policy-Distillation) |
[
Models](https://huggingface.co/collections/wh-zhu/psft-68afb02eb237149f9bd9075e)
Hybrid Policy Distillation (HPD) is a practical distillation framework for reasoning-oriented language models. This repository contains the code, configurations, and experiment assets for the project.
## 🎯 Overview
HPD is designed to make policy distillation more stable and efficient in post-training settings. The current release focuses on:
- 🧭 A unified view of policy distillation methods
- ⚡ Efficient one-hot-style distillation
- 🧩 A hybrid KL objective with a masking mechanism
- 🪶 Lightweight sampling under an offline-prefix setting
## 🗺️ Roadmap
- Release a reproducible Docker image
- Extend HPD to mid-training settings
## 📊 Results
We currently provide a Qwen2.5-1.5B student distilled from Qwen2.5-7B-Thinking with HPD. The released models are available from the Hugging Face collection linked above.
### 🏆 Benchmark Performance
| Model | AIME24 | AIME25 | AMC | MATH | OlympiadMath | GPQA |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Qwen2.5-7B-Thinking
Teacher: [`wh-zhu/Qwen2.5-7B-PSFT-RL-DAPO-90`](https://huggingface.co/wh-zhu/Qwen2.5-7B-PSFT-RL-DAPO-90) | 28.13 | 27.19 | 71.72 | 87.48 | 58.50 | 43.43 |
| Qwen2.5-1.5B-Thinking
Student: [`wh-zhu/qwen2.5-1.5B-longcot-reasoning-HPD`](https://huggingface.co/wh-zhu/qwen2.5-1.5B-longcot-reasoning-HPD) | 7.71 | 9.89 | 39.84 | 63.40 | 32.53 | 28.09 |
### 📈 Training Dynamics
Representative training curves are shown below:
## 🛠️ Implementations
This repository currently provides two implementations:
| Implementation | Location | Training Regime | Entry Point | Notes |
| --- | --- | --- | --- | --- |
| LlamaFactory | `LlamaFactory/` | SFT with HPD loss | `llamafactory-cli train ...` | Supports both full fine-tuning and LoRA |
| verl | `verl/recipe/HPD/` | RL-style / post-training recipe | `bash recipe/HPD/run_hpd.sh` | Current recipe targets `fsdp` / `fsdp2` |
🔍 Key files
### 🦙 LlamaFactory
The LlamaFactory implementation integrates HPD directly into the SFT workflow.
- `LlamaFactory/src/llamafactory/train/hpd.py`: HPD loss implementation
- `LlamaFactory/src/llamafactory/train/sft/trainer.py`: trainer-side HPD integration
- `LlamaFactory/src/llamafactory/hparams/finetuning_args.py`: HPD-related arguments such as `use_hpd_loss`
- `LlamaFactory/examples/train_full/qwen3_full_hpd.yaml`: full-parameter HPD example config
- `LlamaFactory/tests/train/test_sft_trainer.py`: regression test covering HPD loss logging
### ⚙️ verl
The `verl` implementation keeps HPD as a recipe-local extension instead of patching the core PPO workers.
- `verl/recipe/HPD/main_hpd.py`: recipe entry point
- `verl/recipe/HPD/hpd_trainer.py`: HPD trainer built on top of `RayPPOTrainer`
- `verl/recipe/HPD/dp_actor.py`: HPD actor logic
- `verl/recipe/HPD/fsdp_workers.py`: FSDP worker integration
- `verl/recipe/HPD/config/hpd_trainer.yaml`: base Hydra config
- `verl/recipe/HPD/run_hpd.sh`: runnable experiment script
## 📦 Installation
### 🦙 LlamaFactory Backend
Run the following inside `LlamaFactory/`:
```bash
cd LlamaFactory
pip install -e .
pip install -r requirements/metrics.txt
```
If you plan to run full-parameter HPD with DeepSpeed:
```bash
pip install -r requirements/deepspeed.txt
```
For more environment details, see [LlamaFactory/README.md](LlamaFactory/README.md).
### ⚙️ verl Backend
Run the following inside `verl/`:
```bash
cd verl
pip install -e .
```
Depending on your backend, you may also need optional dependencies such as `vllm`, `flash-attn`, or math-evaluation packages. For backend-specific environment setup, see [verl/README.md](verl/README.md).
## 🚀 Usage
### 🦙 LlamaFactory: Full-Parameter HPD
```bash
cd LlamaFactory
llamafactory-cli train examples/train_full/qwen3_full_hpd.yaml
```
This example uses:
- `stage: sft`
- `finetuning_type: full`
- `use_hpd_loss: true`
If `ref_model` is not set, full-parameter HPD uses a frozen copy of `model_name_or_path` as the teacher model.
To use an external teacher, set:
```yaml
ref_model: path_or_hf_repo_of_teacher
ref_model_adapters: path_to_teacher_adapter
ref_model_quantization_bit: 4
```
### 🎛️ LlamaFactory: LoRA HPD
LoRA is also supported, but unlike full fine-tuning, LoRA HPD requires an explicit teacher:
```bash
cd LlamaFactory
llamafactory-cli train examples/train_lora/qwen3_lora_sft.yaml \
use_hpd_loss=true \
ref_model=Qwen/Qwen3-4B-Instruct-2507 \
output_dir=saves/qwen3-4b/lora/hpd
```
### 📝 LlamaFactory Notes
- `use_hpd_loss` is only valid for `stage: sft`
- LoRA HPD requires `ref_model`; full-parameter HPD does not
- Full-parameter HPD keeps both student and teacher in memory, so GPU memory usage is much higher than standard SFT
- The provided full example uses DeepSpeed ZeRO-3 by default for that reason
- If you only want to test the pipeline, reduce `max_samples`, `cutoff_len`, and batch size first
### ⚙️ verl: Quick Start
```bash
cd verl
bash recipe/HPD/run_hpd.sh
```
Before launching the provided script, update the local paths in `recipe/HPD/run_hpd.sh` to match your environment, especially:
- `MODEL_PATH`
- `REF_MODEL_PATH`
- `TRAIN_FILE`
- Evaluation parquet files
- `CKPTS_DIR`
The current recipe uses Hydra configs under `verl/recipe/HPD/config/` and is set up for FSDP-based training.
## 📚 Citation
If you use this repository or the HPD idea in your work, please cite the paper linked above.