{"id":19035735,"url":"https://github.com/woooodyy/llm-reverse-curriculum-rl","last_synced_at":"2025-08-12T16:39:53.163Z","repository":{"id":221552134,"uuid":"754690615","full_name":"WooooDyy/LLM-Reverse-Curriculum-RL","owner":"WooooDyy","description":"Implementation of the ICML 2024 paper \"Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning\" presented by Zhiheng Xi et al.","archived":false,"fork":false,"pushed_at":"2024-02-09T11:35:06.000Z","size":2612,"stargazers_count":97,"open_issues_count":1,"forks_count":6,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-07T14:11:12.270Z","etag":null,"topics":["llm","reasoning","rl"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2402.05808","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/WooooDyy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-08T15:28:31.000Z","updated_at":"2025-04-03T07:40:11.000Z","dependencies_parsed_at":"2024-11-08T22:03:31.090Z","dependency_job_id":null,"html_url":"https://github.com/WooooDyy/LLM-Reverse-Curriculum-RL","commit_stats":null,"previous_names":["woooodyy/llm-reverse-curriculum-rl"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/WooooDyy/LLM-Reverse-Curriculum-RL","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WooooDyy%2FLLM-Reverse-Curriculum-RL","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WooooDyy%2FLLM-Reverse-Curriculum-RL/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WooooDyy%2FLLM-Reverse-Curriculum-RL/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WooooDyy%2FLLM-Reverse-Curriculum-RL/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/WooooDyy","download_url":"https://codeload.github.com/WooooDyy/LLM-Reverse-Curriculum-RL/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WooooDyy%2FLLM-Reverse-Curriculum-RL/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270099211,"owners_count":24527027,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-12T02:00:09.011Z","response_time":80,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["llm","reasoning","rl"],"created_at":"2024-11-08T21:51:56.845Z","updated_at":"2025-08-12T16:39:53.134Z","avatar_url":"https://github.com/WooooDyy.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\n\u003ch1 align=\"left\"\u003e\u003cstrong\u003eR\u003c/strong\u003e\u003csup\u003e3\u003c/sup\u003e: Training Large Language Models for \u003cstrong\u003eR\u003c/strong\u003eeasoning through \u003cstrong\u003eR\u003c/strong\u003eeverse Curriculum \u003cstrong\u003eR\u003c/strong\u003eeinforcement Learning\u003c/h1\u003e\n\n\u003cspace for arxiv badge\u003e\nImplementation of the \"Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning\" presented by Zhiheng Xi, Wenxiang Chen, Boyang Hong, et al.\n\nPaper Link: https://arxiv.org/abs/2402.05808\n## 💡 Introduction\n\n![](src/figures/main.png)\n\n## 🛠️ Set up\n\nIt is suggested to use a **python 3.9** environment to run the experiment. Run the following commands to set up your environment:\n\n```\ngit clone https://github.com/xxxxx.git\n\nconda create -n R3_math python=3.9 -y\ncd R3_math/\npip install -r requirements.txt\n\nconda create -n R3_others python=3.9 -y\ncd R3_others/\npip install -r requirements.txt\n```\n\n## ⚡️Usage\n\n### Step1: SFT Training\n\nTo train a sft model, first set the model path and output path in the  `R3_others/scripts/step1_supervised_finetuning/R3_sft.sh`script. Then, run the following command:\n\n```\ncd R3_others/scripts/step1_supervised_finetuning/\nbash R3_sft.sh\n```\n\n### Step2: R\u003csup\u003e3\u003c/sup\u003e Training\n\nTo train a reinforced model using **R**$^3$ on GSM8K (or other math datasets), first set the actor model path (it should be a sft model checkpoint from **Step1**) and output path in `R3_math/scripts/R3_cot_gsm8k.sh`, and run the following command:\n\n```\ncd R3_math/scripts/\nbash R3_cot_gsm8k.sh\n```\n\n**Note**: If you want to try **R**$^3$ on other datasets like MNLI or race@High, set the SFT model path in `R3_others/scripts/step3_rlhf_finetuning/R3_mix.sh`. Then, run the folloing command:\n\n```\ncd R3_others/scripts/step3_rlhf_finetuning/\nbash R3_mix.sh\n```\n\n### Evaluation\n\n\u003e It is not required for math datasets. Results will be saved in *wandb*.\n\nTo evaluate the model performance, first run the evaluation script `R3_others/scripts/eval/eval_single.sh`. Then, get your results in `output_{dataset_name}.py`. Here's an example for MNLI dataset:\n\n```\ncd R3_others/scripts/eval\nbash eval_single.sh\n# after evaluation\n# you will get a result file like: eval_mnli/R3_test.txt\n\npython output_mnli.py\n# then you will get acc result\n```\n\n### Data\n\nFor the purpose of security review, we provide some examples of the data, formatted as follows:\n\n```\nDataset: MNLI\n\t---- mnli_train_example.json # for SFT\n\t---- mnli_mix_example.json # fot R^3\n\t---- mnli_test.json\n```\n\n## ✏️ Citation\n\nIf you find **R**$^3$ useful for your your research and applications, please cite using this BibTeX:\n\n```\n@misc{xi2024training,\n      title={Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning}, \n      author={Zhiheng Xi and Wenxiang Chen and Boyang Hong and Senjie Jin and Rui Zheng and Wei He and Yiwen Ding and Shichun Liu and Xin Guo and Junzhe Wang and Honglin Guo and Wei Shen and Xiaoran Fan and Yuhao Zhou and Shihan Dou and Xiao Wang and Xinbo Zhang and Peng Sun and Tao Gui and Qi Zhang and Xuanjing Huang},\n      year={2024},\n      eprint={2402.05808},\n      archivePrefix={arXiv},\n      primaryClass={cs.AI}\n}\n```\n\n\n## Contact\n- zhxi22@m.fudan.edu.cn\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwoooodyy%2Fllm-reverse-curriculum-rl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwoooodyy%2Fllm-reverse-curriculum-rl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwoooodyy%2Fllm-reverse-curriculum-rl/lists"}