{"id":20663777,"url":"https://github.com/vita-group/l2o-training-techniques","last_synced_at":"2025-04-19T15:56:52.124Z","repository":{"id":107046198,"uuid":"296398889","full_name":"VITA-Group/L2O-Training-Techniques","owner":"VITA-Group","description":"[NeurIPS 2020 Spotlight Oral] \"Training Stronger Baselines for Learning to Optimize\", Tianlong Chen*, Weiyi Zhang*, Jingyang Zhou, Shiyu Chang, Sijia Liu, Lisa Amini, Zhangyang Wang","archived":false,"fork":false,"pushed_at":"2021-12-30T08:46:56.000Z","size":9445,"stargazers_count":26,"open_issues_count":1,"forks_count":7,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-29T09:51:12.135Z","etag":null,"topics":["curriculum-learning","imitation-learning","learning-to-learn","learning-to-optimize","meta-learning","self-improving","training-tricks"],"latest_commit_sha":null,"homepage":"https://tianlong-chen.github.io/about/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/VITA-Group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-09-17T17:39:28.000Z","updated_at":"2023-12-19T05:45:32.000Z","dependencies_parsed_at":"2023-04-13T15:47:34.156Z","dependency_job_id":null,"html_url":"https://github.com/VITA-Group/L2O-Training-Techniques","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FL2O-Training-Techniques","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FL2O-Training-Techniques/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FL2O-Training-Techniques/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FL2O-Training-Techniques/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/VITA-Group","download_url":"https://codeload.github.com/VITA-Group/L2O-Training-Techniques/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249731679,"owners_count":21317343,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["curriculum-learning","imitation-learning","learning-to-learn","learning-to-optimize","meta-learning","self-improving","training-tricks"],"created_at":"2024-11-16T19:19:48.778Z","updated_at":"2025-04-19T15:56:52.117Z","avatar_url":"https://github.com/VITA-Group.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Training Stronger Baselines for Learning to Optimize\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)\n\nCode for this paper [Training Stronger Baselines for Learning to Optimize]().\n\nTianlong Chen*, Weiyi Zhang*, Jingyang Zhou, Shiyu Chang, Sijia Liu, Lisa Amini, Zhangyang Wang\n\n\n\n## Overview\n\nWith many efforts devoted to designing more sophisticated **L2O models**, we argue for another orthogonal, under-explored theme: the **training techniques** for those L2O models. We show that even **the simplest L2O model could have been trained much better**.\n\n- **Curriculum Learning**\n\n  We first present a progressive training scheme to gradually increase the optimizer unroll length, to mitigate a well-known L2O dilemma of truncation bias (shorter unrolling) versus gradient explosion (longer unrolling).\n\n- **Imitation Learning**\n\n  We further leverage off-policy imitation learning to guide the L2O learning , by taking reference to the behavior of analytical optimizers. \n\nOur improved training techniques are plugged into a variety of state-of-the-art L2O models, and immediately boost their performance, **without making any change to their model structures.** \n\n\n\n## Experiment Results\n\n### Training the L2O-DM baseline to surpass the state-of-the-art\n\n![](./Figs/L2O-DM.png)\n\n### Training state-of-the-art L2O models to boost more performance\n\n![](./Figs/L2O-Scale.png)\n\n### Ablation study of our proposed techniques\n\n![](./Figs/Ablation.png)\n\n### Imitation Learning v.s. Self-Improving\n\n\u003cimg src = \"./Figs/ILvsSelf.png\" align = \"center\" width=\"60%\" hight=\"12%\"\u003e\n\n\n\n## Reproduce Details\n\nExperimental details on L2O-DM and RNNProp are refer to this [README](https://github.com/Tianlong-Chen/L2O-Training-Techniques/blob/master/L2O-DM%20%26%20RNNProp/README.md).\n\nExperimental details on L2O-Scale are refer to this [README](https://github.com/Tianlong-Chen/L2O-Training-Techniques/blob/master/L2O-Scale/README.md).\n\n## Citation\n\nIf you use this code for your research, please cite our paper:\n\n```\n\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvita-group%2Fl2o-training-techniques","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvita-group%2Fl2o-training-techniques","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvita-group%2Fl2o-training-techniques/lists"}