https://github.com/apple/ml-ssd

https://github.com/apple/ml-ssd

Last synced: about 1 month ago
JSON representation

• Host: GitHub
• URL: https://github.com/apple/ml-ssd
• Owner: apple
• License: other
• Created: 2026-03-30T21:14:58.000Z (about 1 month ago)
• Default Branch: main
• Last Pushed: 2026-04-03T23:38:05.000Z (about 1 month ago)
• Last Synced: 2026-04-05T20:02:26.594Z (about 1 month ago)
• Language: Python
• Size: 327 KB
• Stars: 416
• Watchers: 7
• Forks: 35
• Open Issues: 2
• Metadata Files:
  • Readme: README.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# Simple Self-Distillation



[![arXiv](https://img.shields.io/badge/arXiv-2604.01193-b31b1b.svg)](https://arxiv.org/abs/2604.01193)

[![License](https://img.shields.io/badge/License-Apple-blue)](LICENSE)

[![Python](https://img.shields.io/badge/Python-3.10+-green.svg)](https://www.python.org/)

### Embarrassingly Simple Self-Distillation Improves Code Generation

Ruixiang Zhang\*, Richard He Bai\*, Huangjie Zheng\*, Navdeep Jaitly, Ronan Collobert, Yizhe Zhang\*

\*Equal contribution





  



## ✨ Overview

This repository reproduces the method from the paper:

> **Embarrassingly Simple Self-Distillation Improves Code Generation**

The approach consists of three simple steps:

1. **Sample** solutions from a frozen model at non-unit temperature  

2. **Fine-tune** on raw, unverified outputs using standard cross-entropy  

3. **Decode** with a separately tuned temperature  

**No rewards · No verifier · No teacher · No RL**

For full details, see the [paper](https://arxiv.org/abs/2604.01193).

---

## 📰 News

- **[2026-04-03]** 🚀 Initial release of repository  

- **[2026-04-03]** 🤗 Model checkpoints coming soon on Hugging Face  

- *(More updates will be added here)*

---

## 🚀 Getting Started

```bash

git clone https://github.com/apple/ml-ssd.git

cd ml-ssd

uv sync --group evaluation

```

Evaluation commands

```bash

source .venv/bin/activate

python evaluation/eval.py \

    --model  \

    --tensor_parallel_size 4 \

    --max_tokens 65536 \

    --n_repeat 10 \

    --sampling_params "temperature=0.9,top_p=0.8,top_k=20" \

    --output_path ./results/

```

> **Note:** The sampling parameters above are illustrative. Please refer to each model's HuggingFace model card for the recommended sampling parameters.

## 🤗 Models

> Note: Model checkpoints are coming soon. Stay tuned!

## 📁 Repository Structure

```

├── evaluation/

│   ├── eval.py                  # CLI entry point

│   ├── benchmark.py             # LiveCodeBench v6 implementation

│   └── livecodebench_utils.py   # Code execution utilities

├── figures/

│   └── fig_teaser.png

├── pyproject.toml

└── README.md

```

## 📝 Citation

```bibtex

@misc{zhang2026embarrassinglysimpleselfdistillationimproves,

      title={Embarrassingly Simple Self-Distillation Improves Code Generation},

      author={Ruixiang Zhang and Richard He Bai and Huangjie Zheng and Navdeep Jaitly and Ronan Collobert and Yizhe Zhang},

      year={2026},

      eprint={2604.01193},

      archivePrefix={arXiv},

      primaryClass={cs.CL},

      url={https://arxiv.org/abs/2604.01193},

}

```