{"id":25971818,"url":"https://github.com/ai4co/rl4co","last_synced_at":"2026-01-12T02:58:52.984Z","repository":{"id":175393388,"uuid":"605385198","full_name":"ai4co/rl4co","owner":"ai4co","description":"A PyTorch library for all things Reinforcement Learning (RL) for Combinatorial Optimization (CO)","archived":false,"fork":false,"pushed_at":"2024-05-22T20:00:41.000Z","size":54029,"stargazers_count":274,"open_issues_count":12,"forks_count":54,"subscribers_count":7,"default_branch":"main","last_synced_at":"2024-05-22T21:54:47.086Z","etag":null,"topics":["attention","attention-model","benchmark","combinatorial-optimization","cvrp","electronic-design-automation","hydra","neural-combinatorial-optimization","operations-research","pytorch-lightning","reinforcement-learning","tensordict","torchrl","tsp"],"latest_commit_sha":null,"homepage":"https://rl4.co","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ai4co.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-23T03:18:41.000Z","updated_at":"2024-05-27T12:12:50.063Z","dependencies_parsed_at":"2023-11-22T06:23:23.019Z","dependency_job_id":"2705f185-4c32-4ad1-b18f-f96548e04cef","html_url":"https://github.com/ai4co/rl4co","commit_stats":{"total_commits":804,"total_committers":16,"mean_commits":50.25,"dds":"0.19029850746268662","last_synced_commit":"34fe7a38080b34f0ebb643bc5f7a6fde427d065d"},"previous_names":["kaist-silab/rl4co","ai4co/rl4co"],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai4co%2Frl4co","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai4co%2Frl4co/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai4co%2Frl4co/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai4co%2Frl4co/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ai4co","download_url":"https://codeload.github.com/ai4co/rl4co/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241940572,"owners_count":20045881,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention","attention-model","benchmark","combinatorial-optimization","cvrp","electronic-design-automation","hydra","neural-combinatorial-optimization","operations-research","pytorch-lightning","reinforcement-learning","tensordict","torchrl","tsp"],"created_at":"2025-03-05T00:02:02.140Z","updated_at":"2026-01-12T02:58:52.978Z","avatar_url":"https://github.com/ai4co.png","language":"Python","funding_links":[],"categories":["Industry Strength Reinforcement Learning"],"sub_categories":[],"readme":"\u003c/br\u003e\n\u003cdiv align=\"center\"\u003e\n\n\u003cimg src=\"https://raw.githubusercontent.com/ai4co/assets/main/svg/rl4co_animated_full.svg\" alt=\"AI4CO Logo\" style=\"width: 40%; height: auto;\"\u003e\n\n\u003c/br\u003e\u003c/br\u003e\n\n\n\u003ca href=\"https://pytorch.org/get-started/locally/\"\u003e\u003cimg alt=\"PyTorch\" src=\"https://img.shields.io/badge/PyTorch-ee4c2c?logo=pytorch\u0026logoColor=white\"\u003e\u003c/a\u003e\n\u003ca href=\"https://pytorchlightning.ai/\"\u003e\u003cimg alt=\"Lightning\" src=\"https://img.shields.io/badge/-Lightning-792ee5?logo=pytorchlightning\u0026logoColor=white\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/pytorch/rl\"\u003e\u003cimg alt=\"base: TorchRL\" src=\"https://img.shields.io/badge/base-TorchRL-red\"\u003e\u003c/a\u003e\n\u003ca href=\"https://hydra.cc/\"\u003e\u003cimg alt=\"config: Hydra\" src=\"https://img.shields.io/badge/config-Hydra-89b8cd\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/psf/black\"\u003e\u003cimg alt=\"Code style: black\" src=\"https://img.shields.io/badge/code%20style-black-000000.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://join.slack.com/t/ai4co-community/shared_invite/zt-3jsdjs3ec-3KHdV3HwanL884mq_9tyYw\"\u003e\u003cimg alt=\"Slack\" src=\"https://img.shields.io/badge/slack-chat-611f69.svg?logo=slack\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opensource.org/licenses/MIT\"\u003e\u003cimg alt=\"License: MIT\" src=\"https://img.shields.io/badge/License-MIT-red.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://colab.research.google.com/github/ai4co/rl4co/blob/main/examples/1-quickstart.ipynb\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"\u003e\u003c/a\u003e\n\u003ca href=\"https://pypi.org/project/rl4co\"\u003e\u003cimg alt=\"PyPI\" src=\"https://img.shields.io/pypi/v/rl4co?logo=pypi\"\u003e\u003c/a\u003e\n\u003ca href=\"https://app.codecov.io/github/ai4co/rl4co/tree/main/rl4co\"\u003e\u003cimg alt=\"Codecov\" src=\"https://codecov.io/github/ai4co/rl4co/tree/main/badge.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/ai4co/rl4co/actions/workflows/tests.yml\"\u003e\u003cimg alt=\"Test\" src=\"https://github.com/ai4co/rl4co/actions/workflows/tests.yml/badge.svg\"\u003e\u003c/a\u003e\n\n\u003cp\u003e\n  \u003ca href=\"https://rl4co.ai4co.org/\"\u003e\u003cstrong\u003eDocumentation\u003c/strong\u003e\u003c/a\u003e |\n  \u003ca href=\"#getting-started\"\u003e\u003cstrong\u003eGetting Started\u003c/strong\u003e\u003c/a\u003e |\n  \u003ca href=\"#usage\"\u003e\u003cstrong\u003eUsage\u003c/strong\u003e\u003c/a\u003e |\n  \u003ca href=\"#contributing\"\u003e\u003cstrong\u003eContributing\u003c/strong\u003e\u003c/a\u003e |\n  \u003ca href=\"https://arxiv.org/abs/2306.17100\"\u003e\u003cstrong\u003ePaper\u003c/strong\u003e\u003c/a\u003e |\n  \u003ca href=\"#join-us\"\u003e\u003cstrong\u003eJoin Us\u003c/strong\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\n\n\u003c/div\u003e\n\n\n\nAn extensive Reinforcement Learning (RL) for Combinatorial Optimization (CO) benchmark. Our goal is to provide a unified framework for RL-based CO algorithms, and to facilitate reproducible research in this field, decoupling the science from the engineering.\n\n\nRL4CO is built upon:\n- [TorchRL](https://github.com/pytorch/rl): official PyTorch framework for RL algorithms and vectorized environments on GPUs\n- [TensorDict](https://github.com/pytorch-labs/tensordict): a library to easily handle heterogeneous data such as states, actions and rewards\n- [PyTorch Lightning](https://github.com/Lightning-AI/lightning): a lightweight PyTorch wrapper for high-performance AI research\n- [Hydra](https://github.com/facebookresearch/hydra): a framework for elegantly configuring complex applications\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/ai4co/rl4co/assets/48984123/0e409784-05a9-4799-b7aa-6c0f76ecf27f\" alt=\"RL4CO-Overview\" style=\"max-width: 90%;\"\u003e\n\u003c/div\u003e\n\nWe offer flexible and efficient implementations of the following policies:\n- **Constructive**: learn to construct a solution from scratch\n  - _Autoregressive (AR)_: construct solutions one step at a time via a decoder\n  - _NonAutoregressive (NAR)_: learn to predict a heuristic, such as a heatmap, to then construct a solution\n- **Improvement**: learn to improve a pre-existing solution\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/ai4co/rl4co/assets/48984123/9e1f32f9-9884-49b9-b6cd-364861cc8fe7\" alt=\"RL4CO-Policy-Overview\" style=\"max-width: 90%;\"\u003e\n\u003c/div\u003e\n\nWe provide several utilities and modularization. For example, we modularize reusable components such as _environment embeddings_ that can easily be swapped to [solve new problems](https://github.com/ai4co/rl4co/blob/main/examples/3-creating-new-env-model.ipynb).\n\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/ai4co/rl4co/assets/48984123/c47a9301-4c9f-43fd-b21f-761abeae9717\" alt=\"RL4CO-Env-Embedding\" style=\"max-width: 90%;\"\u003e\n\u003c/div\u003e\n\n\n## Getting started\n\u003ca href=\"https://colab.research.google.com/github/ai4co/rl4co/blob/main/examples/1-quickstart.ipynb\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"\u003e\u003c/a\u003e\n\nRL4CO is now available for installation on `pip`!\n```bash\npip install rl4co\n```\n\nTo get started, we recommend checking out our [quickstart notebook](examples/1-quickstart.ipynb) or the [minimalistic example](#minimalistic-example) below.\n\n### Install from source\nThis command installs the bleeding edge `main` version, useful for staying up-to-date with the latest developments - for instance, if a bug has been fixed since the last official release but a new release hasn’t been rolled out yet:\n\n```bash\npip install -U git+https://github.com/ai4co/rl4co.git\n```\n\n### Local install and development\n\nWe recommend local development with the blazing-fast [`uv`](https://docs.astral.sh/uv/) package manager, for instance:\n\n```bash\ngit clone https://github.com/ai4co/rl4co \u0026\u0026 cd rl4co\nuv sync --all-extras\nsource .venv/bin/activate\n```\n\nThis will create a new virtual environment in `.venv/` and install all dependencies.\n\n## Usage\n\n\nTrain model with default configuration (AM on TSP environment):\n```bash\npython run.py\n```\n\n\u003e [!TIP]\n\u003e You may check out [this notebook](examples/advanced/1-hydra-config.ipynb) to get started with Hydra!\n\n\u003cdetails\u003e\n    \u003csummary\u003eChange experiment settings\u003c/summary\u003e\n\nTrain model with chosen experiment configuration from [configs/experiment/](configs/experiment/)\n```bash\npython run.py experiment=routing/am env=tsp env.num_loc=50 model.optimizer_kwargs.lr=2e-4\n```\nHere you may change the environment, e.g. with `env=cvrp` by command line or by modifying the corresponding experiment e.g. [configs/experiment/routing/am.yaml](configs/experiment/routing/am.yaml).\n\n\u003c/details\u003e\n\n\n\n\n\u003cdetails\u003e\n    \u003csummary\u003eDisable logging\u003c/summary\u003e\n\n```bash\npython run.py experiment=routing/am logger=none '~callbacks.learning_rate_monitor'\n```\nNote that `~` is used to disable a callback that would need a logger.\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n    \u003csummary\u003eCreate a sweep over hyperparameters (-m for multirun)\u003c/summary\u003e\n\n```bash\npython run.py -m experiment=routing/am  model.optimizer.lr=1e-3,1e-4,1e-5\n```\n\u003c/details\u003e\n\n\n\n### Minimalistic Example\n\nHere is a minimalistic example training the Attention Model with greedy rollout baseline on TSP in less than 30 lines of code:\n\n```python\nfrom rl4co.envs.routing import TSPEnv, TSPGenerator\nfrom rl4co.models import AttentionModelPolicy, POMO\nfrom rl4co.utils import RL4COTrainer\n\n# Instantiate generator and environment\ngenerator = TSPGenerator(num_loc=50, loc_distribution=\"uniform\")\nenv = TSPEnv(generator)\n\n# Create policy and RL model\npolicy = AttentionModelPolicy(env_name=env.name, num_encoder_layers=6)\nmodel = POMO(env, policy, batch_size=64, optimizer_kwargs={\"lr\": 1e-4})\n\n# Instantiate Trainer and fit\ntrainer = RL4COTrainer(max_epochs=10, accelerator=\"gpu\", precision=\"16-mixed\")\ntrainer.fit(model)\n```\n\nOther examples can be found on our [documentation](https://rl4co.ai4co.org/examples/1-quickstart/)!\n\n\n### Testing\n\nRun tests with `pytest` from the root directory:\n\n```bash\npytest tests\n```\n\n### Known Bugs\n\nYou may check out the [issues](https://github.com/ai4co/rl4co/issues) and [discussions](https://github.com/ai4co/rl4co/discussions). We will also periodically post updates on the [FAQ section](https://rl4co.ai4co.org/docs/content/general/faq/).\n\n## Contributing\n\nHave a suggestion, request, or found a bug? Feel free to [open an issue](https://github.com/ai4co/rl4co/issues) or [submit a pull request](https://github.com/ai4co/rl4co/pulls).\nIf you would like to contribute, please check out our contribution guidelines   [here](.github/CONTRIBUTING.md). We welcome and look forward to all contributions to RL4CO!\n\nWe are also on [Slack](https://join.slack.com/t/ai4co-community/shared_invite/zt-3jsdjs3ec-3KHdV3HwanL884mq_9tyYw) if you have any questions or would like to discuss RL4CO with us. We are open to collaborations and would love to hear from you 🚀\n\n### Contributors\n\u003ca href=\"https://github.com/ai4co/rl4co/graphs/contributors\"\u003e\n  \u003cimg src=\"https://contrib.rocks/image?repo=ai4co/rl4co\" /\u003e\n\u003c/a\u003e\n\n## Citation\nIf you find RL4CO valuable for your research or applied projects:\n\n```bibtex\n@inproceedings{berto2025rl4co,\n    title={{RL4CO: an Extensive Reinforcement Learning for Combinatorial Optimization Benchmark}},\n    author={Federico Berto and Chuanbo Hua and Junyoung Park and Laurin Luttmann and Yining Ma and Fanchen Bu and Jiarui Wang and Haoran Ye and Minsu Kim and Sanghyeok Choi and Nayeli Gast Zepeda and Andr\\'e Hottung and Jianan Zhou and Jieyi Bi and Yu Hu and Fei Liu and Hyeonah Kim and Jiwoo Son and Haeyeon Kim and Davide Angioni and Wouter Kool and Zhiguang Cao and Jie Zhang and Kijung Shin and Cathy Wu and Sungsoo Ahn and Guojie Song and Changhyun Kwon and Lin Xie and Jinkyoo Park},\n    booktitle={Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining},\n    year={2025},\n    url={https://github.com/ai4co/rl4co}\n}\n```\n\nNote that a [previous version of RL4CO](https://openreview.net/forum?id=YXSJxi8dOV) was also accepted as an oral presentation at the [NeurIPS 2023 GLFrontiers Workshop](https://glfrontiers.github.io/). Since then, the library has greatly evolved and improved!\n\n---\n\n\n## Join us\n[![Slack](https://img.shields.io/badge/slack-chat-611f69.svg?logo=slack)](https://join.slack.com/t/ai4co-community/shared_invite/zt-3jsdjs3ec-3KHdV3HwanL884mq_9tyYw)\n\nWe invite you to join our AI4CO community, an open research group in Artificial Intelligence (AI) for Combinatorial Optimization (CO)!\n\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/ai4co/assets/main/svg/ai4co_animated_full.svg\" alt=\"AI4CO Logo\" style=\"width: 30%; height: auto;\"\u003e\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fai4co%2Frl4co","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fai4co%2Frl4co","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fai4co%2Frl4co/lists"}