{"id":27233061,"url":"https://github.com/BytedTsinghua-SIA/DAPO","last_synced_at":"2025-04-10T14:11:21.302Z","repository":{"id":282901587,"uuid":"950036175","full_name":"BytedTsinghua-SIA/DAPO","owner":"BytedTsinghua-SIA","description":"An Open-source RL System from ByteDance Seed and Tsinghua AIR","archived":false,"fork":false,"pushed_at":"2025-04-08T11:26:42.000Z","size":3261,"stargazers_count":1040,"open_issues_count":12,"forks_count":45,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-04-08T12:30:20.810Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BytedTsinghua-SIA.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-17T14:29:24.000Z","updated_at":"2025-04-08T11:43:28.000Z","dependencies_parsed_at":"2025-04-08T12:35:47.085Z","dependency_job_id":null,"html_url":"https://github.com/BytedTsinghua-SIA/DAPO","commit_stats":null,"previous_names":["bytedtsinghua-sia/dapo"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BytedTsinghua-SIA%2FDAPO","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BytedTsinghua-SIA%2FDAPO/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BytedTsinghua-SIA%2FDAPO/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BytedTsinghua-SIA%2FDAPO/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BytedTsinghua-SIA","download_url":"https://codeload.github.com/BytedTsinghua-SIA/DAPO/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248232420,"owners_count":21069487,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-10T14:11:18.808Z","updated_at":"2025-04-10T14:11:21.288Z","avatar_url":"https://github.com/BytedTsinghua-SIA.png","language":null,"funding_links":[],"categories":["Projects","News","4. 算法","Python"],"sub_categories":["Large Language Models","4.2 Reinforcement Learning"],"readme":"\u003cdiv align='center'\u003e\n\u003ch1\u003eDAPO: an Open-source RL System from \u003cbr\u003eByteDance Seed and Tsinghua AIR\u003c/h1\u003e\n\n\u003c!-- TODO:  Thread,Paper,Dataset,Weights--\u003e\n[![Paper](https://img.shields.io/badge/paper-5f16a8?style=for-the-badge\u0026logo=arxiv\u0026logoColor=white)](https://arxiv.org/pdf/2503.14476)\n[![Blog](https://img.shields.io/badge/Blog-3858bf?style=for-the-badge\u0026logo=homepage\u0026logoColor=white)](https://DAPO-SIA.github.io/)\n[![Dataset](https://img.shields.io/badge/Datasets-4d8cd8?style=for-the-badge\u0026logo=huggingface\u0026logoColor=white)](https://huggingface.co/datasets/BytedTsinghua-SIA/DAPO-Math-17k)\n[![Weights](https://img.shields.io/badge/Model%20Weights-63cad3?style=for-the-badge\u0026logo=huggingface\u0026logoColor=white)](https://huggingface.co/BytedTsinghua-SIA/DAPO-Qwen-32B)\n\u003c!-- [![Thread](https://img.shields.io/badge/Thread-91ded6?style=for-the-badge\u0026logo=x\u0026logoColor=white)](https://github.com/BytedTsinghua-SIA/DAPO) --\u003e\n\u003c/div\u003e\n\n\u003e [!IMPORTANT]\n\u003e **🔥 News!!!**\n\u003e - [2025/03] We release the training record of an early version of DAPO (w/o Token-level PG Loss \u0026 Dynamic Sampling), achieving 44% on AIME 2024, in [wandb](https://wandb.ai/verl-org/DAPO%20Reproduction%20on%20verl?nw=u7n2j5sht28).\n\nWe release a fully open-sourced system for large-scale LLM RL, including algorithm, code infrastructure, and dataset. The system achieves state-of-the-art large-scale LLM RL performance. We propose the **D**ecoupled Clip and **D**ynamic s**A**mpling **P**olicy **O**ptimization (**DAPO**) algorithm.\nThrough open-sourcing, we provide the broader research community and society with practical access to scalable reinforcement learning, enabling all to benefit from these advancements. Our system is based on the awesome [verl](https://github.com/volcengine/verl) framework. Thanks for their great work!\n\n## Discussions Welcomed\n\n🤗 If you have any questions about our paper, issues are welcomed and we could discuss there. Thank you!\n\n## Key Results\n\n### AIME 2024 Performance\n\n🚀 **DAPO** achieves 50 points on AIME 2024 based on the Qwen2.5-32B base model, outperforming the previous SoTA DeepSeek-R1-Zero-Qwen-32B with 50% training steps.\n\n![alt text](img/score.png)\n\n### Metric Supervision during Training\n\n1. **Length stability and growth**: The steady increase in response length allows for greater exploration, facilitating the model’s ability to learn more complex reasoning behaviors, ultimately contributing to training stability and performance improvement.\n\n2. **Reward score stability**: A stable increase in the reward signal indicates that the model is successfully fitting the training distribution, ensuring that the learning process remains robust and consistent without significant fluctuations.\n\n3. **Entropy and mean probability trend**: A controlled increase in entropy, after an initial decrease, ensures a healthy balance between exploration and exploitation, avoiding issues such as overfitting or excessive randomness, and promoting sustained model performance.\n\n![alt text](img/dynamic.png)\n\n## Reproducibility\n\nTo benefit the broader research community, we fully open-source the recipe of our RL training, including algorithm details, dataset, and infrastructures.\n\n### Datasets\nWe provide training and validation datasets for DAPO training.\n\nTraining: [DAPO-Math-17k](https://huggingface.co/datasets/BytedTsinghua-SIA/DAPO-Math-17k), a carefully curated and processed math dataset.\nValidation: [AIME 2024](https://huggingface.co/datasets/BytedTsinghua-SIA/AIME-2024).\n\n### Training\n\nWe provide the [out-of-the-box](https://github.com/volcengine/verl/blob/gm-tyx/puffin/main/recipe/dapo) script for DAPO training reproduction. Quickstart and core code are mentioned in [README](https://github.com/volcengine/verl/blob/gm-tyx/puffin/main/recipe/dapo/README.md). These are scripts for:\n\n- [Datasets Preparation](https://github.com/volcengine/verl/blob/gm-tyx/puffin/main/recipe/dapo/prepare_dapo_data.sh)\n- [DAPO w/o Token-level PG Loss \u0026 Dynamic Sampling -- AIME 44](https://github.com/volcengine/verl/blob/gm-tyx/puffin/main/recipe/dapo/run_dapo_early_qwen2.5_32b.sh)\n- [DAPO Full -- AIME 50](https://github.com/volcengine/verl/blob/gm-tyx/puffin/main/recipe/dapo/run_dapo_qwen2.5_32b.sh)\n\nNote:\n\n- The `DAPO w/o Token-level PG Loss \u0026 Dynamic Sampling -- AIME 44` script has been verified on the current verl and achieves 44 points on AIME, whose training record can be accessed in [wandb](https://wandb.ai/verl-org/DAPO%20Reproduction%20on%20verl?nw=u7n2j5sht28).\n\n- The final performance of DAPO (50 on AIME) is achieved using the full DAPO algorithm based on our internal codebase, which includes heavy engineering optimization code based on verl. The `DAPO Full` script provides the command to run the full DAPO algorithm. But we still have not verified it on verl.\n\n## Acknowledgement\n\nWe thank the [verl](https://github.com/volcengine/verl) for providing the awesome open-source RL infrastructure.\n\nOur open-sourced experiments were conducted on the Volcano Engine Machine Learning Platform. We will provide a full reproduction guideline later on the Volcano Engine platform to help users replicate our experiments.\n\n\u003c!-- ## Citation --\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FBytedTsinghua-SIA%2FDAPO","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FBytedTsinghua-SIA%2FDAPO","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FBytedTsinghua-SIA%2FDAPO/lists"}