{"id":17326087,"url":"https://github.com/lucasalegre/morl-baselines","last_synced_at":"2025-05-15T08:10:33.764Z","repository":{"id":65434799,"uuid":"516834101","full_name":"LucasAlegre/morl-baselines","owner":"LucasAlegre","description":"Multi-Objective Reinforcement Learning algorithms implementations.","archived":false,"fork":false,"pushed_at":"2024-12-04T14:54:34.000Z","size":23567,"stargazers_count":368,"open_issues_count":12,"forks_count":61,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-14T15:00:01.409Z","etag":null,"topics":["gym","gymnasium","mo-gymnasium","morl","multi-objective","multi-objective-rl","pytorch","reinforcement-learning","rl","rl-algorithms"],"latest_commit_sha":null,"homepage":"https://lucasalegre.github.io/morl-baselines","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LucasAlegre.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.bib","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-22T17:26:05.000Z","updated_at":"2025-04-14T12:21:09.000Z","dependencies_parsed_at":"2023-11-09T11:49:02.482Z","dependency_job_id":"e28f0fee-ce50-4562-be8e-f7eddc2a0162","html_url":"https://github.com/LucasAlegre/morl-baselines","commit_stats":{"total_commits":391,"total_committers":14,"mean_commits":"27.928571428571427","dds":0.5294117647058824,"last_synced_commit":"92572aa5ee13e98b9e080f64df8e29602dba4f69"},"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LucasAlegre%2Fmorl-baselines","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LucasAlegre%2Fmorl-baselines/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LucasAlegre%2Fmorl-baselines/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LucasAlegre%2Fmorl-baselines/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LucasAlegre","download_url":"https://codeload.github.com/LucasAlegre/morl-baselines/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254301433,"owners_count":22047904,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gym","gymnasium","mo-gymnasium","morl","multi-objective","multi-objective-rl","pytorch","reinforcement-learning","rl","rl-algorithms"],"created_at":"2024-10-15T14:14:58.947Z","updated_at":"2025-05-15T08:10:33.701Z","avatar_url":"https://github.com/LucasAlegre.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Project Status: Active – The project has reached a stable, usable state and is being actively developed.](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active)\n![tests](https://github.com/LucasAlegre/morl-baselines/workflows/Python%20tests/badge.svg)\n[![License](http://img.shields.io/badge/license-MIT-brightgreen.svg?style=flat)](https://github.com/LucasAlegre/morl-baselines/blob/main/LICENSE)\n[![Discord](https://img.shields.io/discord/999693014618362036?label=discord)](https://discord.gg/ygmkfnBvKA)\n[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit\u0026logoColor=white)](https://pre-commit.com/)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n[![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat\u0026labelColor=ef8336)](https://pycqa.github.io/isort/)\n\n\u003cimg src=\"docs/_static/_images/mo_cheetah.gif\" alt=\"Multiple policies\" align=\"right\" width=\"50%\"/\u003e\n\n# MORL-Baselines\n\n\u003c!-- start elevator-pitch --\u003e\n\nMORL-Baselines is a library of Multi-Objective Reinforcement Learning (MORL) algorithms.\nThis repository aims to contain reliable MORL algorithms implementations in PyTorch.\n\nIt strictly follows [MO-Gymnasium](https://github.com/Farama-Foundation/mo-gymnasium) API, which differs from the standard [Gymnasium](https://github.com/Farama-Foundation/Gymnasium) API only in that the environment returns a numpy array as the reward.\n\nFor details on multi-objective MDPs (MOMDPs) and other MORL definitions, we suggest reading [A practical guide to multi-objective reinforcement learning and planning](https://link.springer.com/article/10.1007/s10458-022-09552-y). An overview of some techniques used in various MORL algorithms is also provided in [Multi-Objective Reinforcement Learning Based on Decomposition: A Taxonomy and Framework](https://doi.org/10.1613/jair.1.15702).\n\nA tutorial on MO-Gymnasium and MORL-Baselines is also available: [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1ByjuUp8-CJeh1giPOACqPGiglPxDnlSq?usp=sharing)\n\n\u003c!-- end elevator-pitch --\u003e\n\n## Features\n\n\u003c!-- start features --\u003e\n\n* Single and multi-policy algorithms under both SER and ESR criteria are implemented.\n* All algorithms follow the [MO-Gymnasium](https://www.github.com/Farama-Foundation/mo-gymnasium) API.\n* Performances are automatically reported in [Weights and Biases](https://wandb.ai/) dashboards.\n* Linting and formatting are enforced by pre-commit hooks.\n* Code is well documented.\n* All algorithms are automatically tested.\n* Utility functions are provided e.g. pareto pruning, experience buffers, etc.\n* Performances have been tested and reported in a reproducible manner.\n* Hyperparameter optimization available.\n\n\u003c!-- end features --\u003e\n\n\n## Implemented Algorithms\n\n\u003c!-- start algos-list --\u003e\n\n| **Name**                                                                                                                                                             | Single/Multi-policy | ESR/SER                     | Observation space | Action space          | Paper                                                                                                                                                    |\n|----------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------|-----------------------------|-------------------|-----------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|\n| [GPI-LS + GPI-PD](https://github.com/LucasAlegre/morl-baselines/blob/main/morl_baselines/multi_policy/gpi_pd/gpi_pd.py)                                              | Multi               | SER                         | Continuous        | Discrete / Continuous | [Paper and Supplementary Materials](https://arxiv.org/abs/2301.07784)                                                                                    |\n| [MORL/D](https://github.com/LucasAlegre/morl-baselines/blob/main/morl_baselines/multi_policy/morld/morld.py)                                                         | Multi               | /                           | /                 | /                     | [Paper](https://doi.org/10.1613/jair.1.15702)                                                                                                                |\n| [Envelope Q-Learning](https://github.com/LucasAlegre/morl-baselines/blob/main/morl_baselines/multi_policy/envelope/envelope.py)                                      | Multi               | SER                         | Continuous        | Discrete              | [Paper](https://arxiv.org/pdf/1908.08342.pdf)                                                                                                            |\n| [CAPQL](https://github.com/LucasAlegre/morl-baselines/blob/main/morl_baselines/multi_policy/capql/capql.py)                                                          | Multi               | SER                         | Continuous        | Continuous            | [Paper](https://openreview.net/pdf?id=TjEzIsyEsQ6)                                                                                                       |\n| [PGMORL](https://github.com/LucasAlegre/morl-baselines/blob/main/morl_baselines/multi_policy/pgmorl/pgmorl.py) \u003csup\u003e[1](#f1)\u003c/sup\u003e                                   | Multi               | SER                         | Continuous        | Continuous            | [Paper](https://people.csail.mit.edu/jiex/papers/PGMORL/paper.pdf) / [Supplementary Materials](https://people.csail.mit.edu/jiex/papers/PGMORL/supp.pdf) |\n| [Pareto Conditioned Networks (PCN)](https://github.com/LucasAlegre/morl-baselines/blob/main/morl_baselines/multi_policy/pcn/pcn.py)                                  | Multi               | SER/ESR \u003csup\u003e[2](#f2)\u003c/sup\u003e | Continuous        | Discrete / Continuous | [Paper](https://www.ifaamas.org/Proceedings/aamas2022/pdfs/p1110.pdf)                                                                                    |\n| [Pareto Q-Learning](https://github.com/LucasAlegre/morl-baselines/blob/main/morl_baselines/multi_policy/pareto_q_learning/pql.py)                                    | Multi               | SER                         | Discrete          | Discrete              | [Paper](https://jmlr.org/papers/volume15/vanmoffaert14a/vanmoffaert14a.pdf)                                                                              |\n| [MO Q learning](https://github.com/LucasAlegre/morl-baselines/blob/main/morl_baselines/single_policy/ser/mo_q_learning.py)                                           | Single              | SER                         | Discrete          | Discrete              | [Paper](https://www.researchgate.net/publication/235698665_Scalarized_Multi-Objective_Reinforcement_Learning_Novel_Design_Techniques)                    |\n| [MPMOQLearning](https://github.com/LucasAlegre/morl-baselines/blob/main/morl_baselines/multi_policy/multi_policy_moqlearning/mp_mo_q_learning.py)  (outer loop MOQL) | Multi               | SER                         | Discrete          | Discrete              | [Paper](https://www.researchgate.net/publication/235698665_Scalarized_Multi-Objective_Reinforcement_Learning_Novel_Design_Techniques)                    |\n| [Optimistic Linear Support (OLS)](https://github.com/LucasAlegre/morl-baselines/blob/main/morl_baselines/multi_policy/ols/ols.py)                                    | Multi               | SER                         | /                 | /                     | Section 3.3 of the [thesis](http://roijers.info/pub/thesis.pdf)                                                                                          |\n| [Expected Utility Policy Gradient (EUPG)](https://github.com/LucasAlegre/morl-baselines/blob/main/morl_baselines/single_policy/esr/eupg.py)                          | Single              | ESR                         | Discrete          | Discrete              | [Paper](https://www.researchgate.net/publication/328718263_Multi-objective_Reinforcement_Learning_for_the_Expected_Utility_of_the_Return)                |\n\n:warning: Some of the algorithms have limited features.\n\n\u003cb id=\"f1\"\u003e1\u003c/b\u003e: Currently, PGMORL is limited to environments with 2 objectives.\n\n\u003cb id=\"f2\"\u003e2\u003c/b\u003e: PCN assumes environments with deterministic transitions.\n\n\u003c!-- end algos-list --\u003e\n\n## Benchmarking\n\n\u003c!-- start benchmark --\u003e\nMORL-Baselines participates to [Open RL Benchmark](https://github.com/openrlbenchmark/openrlbenchmark) which contains tracked experiments from popular RL libraries such as [cleanRL](https://github.com/vwxyzjn/cleanrl) and [Stable Baselines 3](https://github.com/DLR-RM/stable-baselines3).\n\n\nWe have run experiments of our algorithms on various environments from [MO-Gymnasium](https://www.github.com/Farama-Foundation/mo-gymnasium). The results can be found here: https://wandb.ai/openrlbenchmark/MORL-Baselines. An issue tracking all the settings is available at [#43](https://github.com/LucasAlegre/morl-baselines/issues/43). Some design documentation for the experimentation protocol are also available on our [Documentation website](https://lucasalegre.github.io/morl-baselines/algos/performances/).\n\u003c!-- end benchmark --\u003e\n\nAn example visualization of our dashboards with Pareto support is shown below:\n\u003cimg src=\"docs/_static/_images/wandb.png\" alt=\"WandB dashboards\"/\u003e\n\n## Structure\n\n\u003c!-- start structure --\u003e\nAs much as possible, this repo tries to follow the single-file implementation rule for all algorithms. The repo's structure is as follows:\n\n* `examples/` contains a set of examples to use MORL Baselines with [MO-Gymnasium](https://www.github.com/Farama-Foundation/mo-gymnasium) environments.\n* `common/` contains the implementation recurring concepts: replay buffers, neural nets, etc. See the [documentation](https://lucasalegre.github.io/morl-baselines/) for more details.\n* `multi_policy/` contains the implementations of multi-policy algorithms.\n* `single_policy/` contains the implementations of single-policy algorithms (ESR and SER).\n\n\u003c!-- end structure --\u003e\n\n\n## Citing the Project\n\n\u003c!-- start citing --\u003e\nIf you use MORL-Baselines in your research, please cite our [NeurIPS 2023 paper](https://openreview.net/pdf?id=jfwRLudQyj):\n\n```bibtex\n@inproceedings{felten_toolkit_2023,\n\tauthor = {Felten, Florian and Alegre, Lucas N. and Now{\\'e}, Ann and Bazzan, Ana L. C. and Talbi, El Ghazali and Danoy, Gr{\\'e}goire and Silva, Bruno Castro da},\n\ttitle = {A Toolkit for Reliable Benchmarking and Research in Multi-Objective Reinforcement Learning},\n\tbooktitle = {Proceedings of the 37th Conference on Neural Information Processing Systems ({NeurIPS} 2023)},\n\tyear = {2023}\n}\n```\n\n\u003c!-- end citing --\u003e\n\n## Maintainers\n\n\u003c!-- start maintainers --\u003e\nMORL-Baselines is currently maintained by [Florian Felten](https://ffelten.github.io/) (@ffelten) and [Lucas N. Alegre](https://www.inf.ufrgs.br/~lnalegre/) (@LucasAlegre).\n\u003c!-- end maintainers --\u003e\n\n## Contributing\n\n\u003c!-- start contributing --\u003e\nThis repository is open to contributions and we are always happy to receive new algorithms, bug fixes, or features. If you want to contribute, you can join our [Discord server](https://discord.gg/ygmkfnBvKA) and discuss your ideas with us. You can also open an issue or a pull request directly.\n\n\u003c!-- end contributing --\u003e\n\n## Acknowledgements\n\u003c!-- start acknowledgements --\u003e\n* Willem Röpke, for his implementation of Pareto Q-Learning (@wilrop)\n* Mathieu Reymond, for providing us with the original implementation of PCN.\n* Denis Steckelmacher and Conor F. Hayes, for providing us with the original implementation of EUPG.\n\u003c!-- end acknowledgements --\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucasalegre%2Fmorl-baselines","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flucasalegre%2Fmorl-baselines","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucasalegre%2Fmorl-baselines/lists"}