{"id":20663711,"url":"https://github.com/vita-group/open-l2o","last_synced_at":"2025-04-09T11:09:34.015Z","repository":{"id":39738704,"uuid":"326295386","full_name":"VITA-Group/Open-L2O","owner":"VITA-Group","description":"Open-L2O: A Comprehensive and Reproducible Benchmark for Learning to Optimize Algorithms","archived":false,"fork":false,"pushed_at":"2023-06-09T02:17:42.000Z","size":18452,"stargazers_count":277,"open_issues_count":3,"forks_count":43,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-02T10:11:54.995Z","etag":null,"topics":["convex-optimization","lasso","learning-to-learn","learning-to-optimize","meta-learning","neural-networks","nonconvex-optimization"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/VITA-Group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-01-03T00:09:21.000Z","updated_at":"2025-04-02T06:44:46.000Z","dependencies_parsed_at":"2025-01-09T10:13:25.889Z","dependency_job_id":"a2c1b6d9-4568-4653-8344-400d23e4777e","html_url":"https://github.com/VITA-Group/Open-L2O","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FOpen-L2O","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FOpen-L2O/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FOpen-L2O/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FOpen-L2O/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/VITA-Group","download_url":"https://codeload.github.com/VITA-Group/Open-L2O/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248027406,"owners_count":21035594,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["convex-optimization","lasso","learning-to-learn","learning-to-optimize","meta-learning","neural-networks","nonconvex-optimization"],"created_at":"2024-11-16T19:19:29.106Z","updated_at":"2025-04-09T11:09:33.997Z","avatar_url":"https://github.com/VITA-Group.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Open-L2O\n\nThis repository establishes the first comprehensive benchmark efforts of existing learning to optimize (L2O) approaches on a number of problems and settings. We release our software implementation and data as the Open-L2O package, for reproducible research and fair benchmarking in the L2O field. [[Paper](https://arxiv.org/abs/2103.12828)]\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)\n\n## Overview\n### What is learning to optimize (L2O)?\n\nL2O (Learning to optimize) aims to replace manually designed analytic optimization algorithms (SGD, RMSProp, Adam, etc.) with learned update rules.\n\n![](Figs/Framework.png)\n\n### How does L2O work?\n\nL2O serves as functions that can be fit from data. L2O gains experience from training optimization tasks in a principled and automatic way.\n\n### What can L2O do for you?\n\nL2O is particularly suitable for solving a certain type of optimization over a specific distribution of data repeatedly. In comparison to classic methods, L2O is shown to find higher-quality solutions and/or with much faster convergence speed for many problems.\n\n### Open questions for research?\n\n- There are significant theoretical and practicality gaps between manually designed optimizers and existing L2O models.\n\n\n## Main Results\n### Learning to optimize sparse recovery\n\n![](Figs/ss.png)\n\n### Learning to optimize Lasso functions\n\n![](Figs/lasso.png)\n\n### Learning to optimize non-convex Rastrigin functions\n\n![](Figs/ras.png)\n\n### Learning to optimize neural networks\n\n![](Figs/nn.png)\n\n\n## Supported Model-base Learnable Optimizers\n\nAll codes are available at [here](https://github.com/VITA-Group/Open-L2O/tree/main/Model_Base_L2O).\n\n1. LISTA (feed-forward form) from *Learning fast approximations of sparse coding* [[Paper](https://dl.acm.org/doi/abs/10.5555/3104322.3104374)]\n2. LISTA-CP from *Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholds* [[Paper](https://arxiv.org/abs/1808.10038)]\n3. LISTA-CPSS from *Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholds* [[Paper](https://arxiv.org/abs/1808.10038)]\n4. LFISTA from *Understanding Trainable Sparse Coding via Matrix Factorization* [[Paper](https://arxiv.org/abs/1609.00285)]\n5. LAMP from *AMP-Inspired Deep Networks for Sparse Linear Inverse Problems* [[Paper](https://ieeexplore.ieee.org/document/7934066)]\n6. ALISTA from *ALISTA: Analytic Weights Are As Good As Learned Weights in LISTA* [[Paper](https://openreview.net/forum?id=B1lnzn0ctQ)]\n7. GLISTA from *Sparse Coding with Gated Learned ISTA* [[Paper](https://openreview.net/forum?id=BygPO2VKPH)]\n\n\n## Supported Model-free Learnable Optimizers\n\n1. L2O-DM from *Learning to learn by gradient descent by gradient descent* [[Paper](https://arxiv.org/pdf/1606.04474.pdf)] [[Code](https://github.com/Tianlong-Chen/Awesome-L2O/blob/main/Model_Free_L2O/L2O-DM%20and%20L2O-RNNProp/README.md)]\n2. L2O-RNNProp *Learning Gradient Descent: Better Generalization and Longer Horizons* from [[Paper](https://arxiv.org/pdf/1703.03633.pdf)] [[Code](https://github.com/Tianlong-Chen/Awesome-L2O/blob/main/Model_Free_L2O/L2O-DM%20and%20L2O-RNNProp/README.md)]\n3. L2O-Scale from *Learned Optimizers that Scale and Generalize* [[Paper](https://arxiv.org/pdf/1703.04813.pdf)] [[Code](https://github.com/Tianlong-Chen/Awesome-L2O/blob/main/Model_Free_L2O/L2O-Scale%20/README.md)]\n4. L2O-enhanced from *Training Stronger Baselines for Learning to Optimize* [[Paper](https://arxiv.org/pdf/2010.09089.pdf)] [[Code](https://github.com/Tianlong-Chen/Awesome-L2O/blob/main/Model_Free_L2O/L2O-DM%20and%20L2O-RNNProp/README.md)]\n5. L2O-Swarm from *Learning to Optimize in Swarms* [[Paper](https://papers.nips.cc/paper/2019/file/ec04e8ebba7e132043e5b4832e54f070-Paper.pdf)] [[Code](https://github.com/Tianlong-Chen/Awesome-L2O/blob/main/Model_Free_L2O/L2O-Swarm/README.md)]\n6. L2O-Jacobian from *HALO: Hardware-Aware Learning to Optimize* [[Paper](http://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123540477.pdf)] [[Code](https://github.com/Tianlong-Chen/Awesome-L2O/blob/main/Model_Free_L2O/L2O-Jacobian/README.md)]\n7. L2O-Minmax from *Learning A Minimax Optimizer: A Pilot Study* [[Paper](https://openreview.net/forum?id=nkIDwI6oO4_)] [[Code]()]\n\n\n\n## Supported Optimizees\n\nConvex Functions:\n\n- [x] Quadratic\n- [x]  Lasso\n\nNon-convex Functions:\n\n- [x] Rastrigin\n\nMinmax Functions:\n\n- [x] Saddle\n- [x] Rotated Saddle\n- [x] Seesaw\n- [x] Matrix Game\n\nNeural Networks:\n\n- [x] MLPs on MNIST\n- [x] ConvNets on MNIST and CIFAR-10\n- [x] LeNet\n- [x] NAS searched archtectures\n\n\n\n## Other Resources\n\n- This is a Pytorch implementation of L2O-DM. [[Code](https://github.com/chenwydj/learning-to-learn-by-gradient-descent-by-gradient-descent)]\n- This is the original L2O-Swarm repository. [[Code](https://github.com/Shen-Lab/LOIS)]\n- This is the original L2O-Jacobian repository. [[Code](https://github.com/RICE-EIC/HALO)]\n\n## Hardware Information\n\nOur experiments were conducted on a cluster with two GPUs (GeForce RTX 3080) and a 14-core CPU (Intel(R) Core(TM) i9-9940X).\n\n## Future Works\n\n- [ ] TF2.0 Implementated toolbox v2 with a unified framework and lib dependency.\n\n\n\n## Cite\n\n```\n@misc{chen2021learning,\n      title={Learning to Optimize: A Primer and A Benchmark}, \n      author={Tianlong Chen and Xiaohan Chen and Wuyang Chen and Howard Heaton and Jialin Liu and Zhangyang Wang and Wotao Yin},\n      year={2021},\n      eprint={2103.12828},\n      archivePrefix={arXiv},\n      primaryClass={math.OC}\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvita-group%2Fopen-l2o","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvita-group%2Fopen-l2o","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvita-group%2Fopen-l2o/lists"}