{"id":13729960,"url":"https://github.com/facebookresearch/mbrl-lib","last_synced_at":"2025-05-08T02:30:59.008Z","repository":{"id":37538141,"uuid":"288181946","full_name":"facebookresearch/mbrl-lib","owner":"facebookresearch","description":"Library for Model Based RL ","archived":true,"fork":false,"pushed_at":"2024-07-12T20:42:46.000Z","size":35262,"stargazers_count":996,"open_issues_count":23,"forks_count":163,"subscribers_count":24,"default_branch":"main","last_synced_at":"2025-04-27T21:03:49.171Z","etag":null,"topics":["model-based-reinforcement-learning","reinforcement-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/facebookresearch.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-08-17T13:03:30.000Z","updated_at":"2025-04-25T01:01:31.000Z","dependencies_parsed_at":"2024-01-06T13:09:18.209Z","dependency_job_id":"eef0c5ab-9f79-4913-b5f2-c99cc80af9e6","html_url":"https://github.com/facebookresearch/mbrl-lib","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fmbrl-lib","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fmbrl-lib/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fmbrl-lib/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fmbrl-lib/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/facebookresearch","download_url":"https://codeload.github.com/facebookresearch/mbrl-lib/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252986655,"owners_count":21836202,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["model-based-reinforcement-learning","reinforcement-learning"],"created_at":"2024-08-03T02:01:08.139Z","updated_at":"2025-05-08T02:30:57.765Z","avatar_url":"https://github.com/facebookresearch.png","language":"Python","funding_links":[],"categories":["Reinforcement Learning (RL) and Deep Reinforcement Learning (DRL)","Codebase"],"sub_categories":["RL/DRL Algorithm Implementations and Software Frameworks","Other"],"readme":"[![PyPi Version](https://img.shields.io/pypi/v/mbrl)](https://pypi.org/project/mbrl/)\n[![Main](https://github.com/facebookresearch/mbrl-lib/workflows/CI/badge.svg)](https://github.com/facebookresearch/mbrl-lib/actions?query=workflow%3ACI)\n[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/facebookresearch/mbrl-lib/tree/main/LICENSE)\n[![Python 3.7+](https://img.shields.io/badge/python-3.7+-blue.svg)](https://www.python.org/downloads/release/python-360/)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n \n\n# MBRL-Lib\n\n``mbrl`` is a toolbox for facilitating development of \nModel-Based Reinforcement Learning algorithms. It provides easily interchangeable \nmodeling and planning components, and a set of utility functions that allow writing\nmodel-based RL algorithms with only a few lines of code. \n\nSee also our companion [paper](https://arxiv.org/abs/2104.10159). \n\n## Getting Started\n\n### Installation\n\n#### Standard Installation\n\n``mbrl`` requires Python 3.8+ library and [PyTorch (\u003e= 1.7)](https://pytorch.org). \nTo install the latest stable version, run\n\n    pip install mbrl\n\n#### Developer installation\nIf you are interested in modifying the library, clone the repository and set up \na development environment as follows\n\n    git clone https://github.com/facebookresearch/mbrl-lib.git\n    pip install -e \".[dev]\"\n\nAnd test it by running the following from the root folder of the repository\n\n    python -m pytest tests/core\n    python -m pytest tests/algorithms\n\n\n### Basic example\nAs a starting point, check out our [tutorial notebook](https://github.com/facebookresearch/mbrl-lib/tree/main/notebooks/pets_example.ipynb) \non how to write the PETS algorithm \n([Chua et al., NeurIPS 2018](https://arxiv.org/pdf/1805.12114.pdf)) \nusing our toolbox, and running it on a continuous version of the cartpole \nenvironment.\n\n## Provided algorithm implementations\nMBRL-Lib provides implementations of popular MBRL algorithms \nas examples of how to use this library. You can find them in the \n[mbrl/algorithms](https://github.com/facebookresearch/mbrl-lib/tree/main/mbrl/algorithms) folder. Currently, we have implemented\n[PETS](https://github.com/facebookresearch/mbrl-lib/tree/main/mbrl/algorithms/pets.py),\n[MBPO](https://github.com/facebookresearch/mbrl-lib/tree/main/mbrl/algorithms/mbpo.py),\n[PlaNet](https://github.com/facebookresearch/mbrl-lib/tree/main/mbrl/algorithms/planet.py), \nwe plan to keep increasing this list in the future.\n\nThe implementations rely on [Hydra](https://github.com/facebookresearch/hydra) \nto handle configuration. You can see the configuration files in \n[this](https://github.com/facebookresearch/mbrl-lib/tree/main/mbrl/examples/conf) \nfolder. \nThe [overrides](https://github.com/facebookresearch/mbrl-lib/tree/main/mbrl/examples/conf/overrides) \nsubfolder contains\nenvironment specific configurations for each environment, overriding the \ndefault configurations with the best hyperparameter values we have found so far \nfor each combination of algorithm and environment. You can run training\nby passing the desired override option via command line. \nFor example, to run MBPO on the [Gymnasium](https://github.com/Farama-Foundation/Gymnasium/) version of HalfCheetah, you should call\n```python\npython -m mbrl.examples.main algorithm=mbpo overrides=mbpo_halfcheetah \n```\nBy default, all algorithms will save results in a csv file called `results.csv`,\ninside a folder whose path looks like \n`./exp/mbpo/default/gym___HalfCheetah-v2/yyyy.mm.dd/hhmmss`; \nyou can change the root directory (`./exp`) by passing \n`root_dir=path-to-your-dir`, and the experiment sub-folder (`default`) by\npassing `experiment=your-name`. The logger will also save a file called \n`model_train.csv` with training information for the dynamics model.\n\nBeyond the override defaults, You can also change other configuration options, \nsuch as the type of dynamics model \n(e.g., `dynamics_model=basic_ensemble`), or the number of models in the ensemble \n(e.g., `dynamics_model.model.ensemble_size=some-number`). To learn more about\nall the available options, take a look at the provided \n[configuration files](https://github.com/facebookresearch/mbrl-lib/tree/main/mbrl/examples/conf). \n\n## Supported environments\nOur example configurations are largely based on [Mujoco](https://mujoco.org/), but\nour library components (and algorithms) are compatible with any environment that follows\nthe standard [Gymnasium](https://github.com/Farama-Foundation/Gymnasium/) syntax. You can try our utilities in other environments \nby creating your own entry script and Hydra configuration, using our default entry \n[`main.py`](https://github.com/facebookresearch/mbrl-lib/blob/main/mbrl/examples/main.py) as guiding template. \nSee also the example [override](https://github.com/facebookresearch/mbrl-lib/tree/main/mbrl/examples/conf/overrides)\nconfigurations. \n\nWithout any modifications, our provided `main.py` can be used to launch experiments with the following environments:\n  * [`mujoco`](https://github.com/deepmind/mujoco)\n  * [`dm_control`](https://github.com/deepmind/dm_control)\n  * [`pybullet-gym`](https://github.com/benelot/pybullet-gym) (thanks to [dtch1997](https://github.com/dtch1997)) for the contribution!\n  Note: You must run `pip install gym==0.26.3` to use the dm_control and pybulletgym environments.\n\nYou can test your Mujoco and PyBullet installations by running\n\n    python -m pytest tests/mujoco\n    python -m pytest tests/pybullet\n\nTo specify the environment to use for `main.py`, there are two possibilities:\n\n  * **Preferred way**: Use a Hydra dictionary to specify arguments for your env constructor. See [example](https://github.com/facebookresearch/mbrl-lib/blob/main/mbrl/examples/conf/overrides/planet_cartpole_balance.yaml#L4).\n  * Less flexible alternative: A single string with the following syntax:\n      - `mujoco-gym`: `\"gym___\u003cenv-name\u003e\"`, where `env-name` is the name of the environment in Gymnasium (e.g., \"HalfCheetah-v2\").\n      - `dm_control`: `\"dmcontrol___\u003cdomain\u003e--\u003ctask\u003e`, where domain/task are defined as in DMControl (e.g., \"cheetah--run\").\n      - `pybullet-gym`: `\"pybulletgym___\u003cenv-name\u003e\"`, where `env-name` is the name of the environment in pybullet gym (e.g., \"HopperPyBulletEnv-v0\")\n\n## Visualization and diagnostics tools\nOur library also contains a set of \n[diagnostics](https://github.com/facebookresearch/mbrl-lib/tree/main/mbrl/diagnostics) tools, meant to facilitate \ndevelopment and debugging of models and controllers. With the exception of the CPU-controller, which also supports \nPyBullet, these currently require a Mujoco installation, but we are planning to add support for other environments \nand extensions in the future. Currently, the following tools are provided:\n\n* ``Visualizer``: Creates a video to qualitatively\nassess model predictions over a rolling horizon. Specifically, it runs a \n  user specified policy in a given environment, and at each time step, computes\n  the model's predicted observation/rewards over a lookahead horizon for the \n  same policy. The predictions are plotted as line plots, one for each \n  observation dimension (blue lines) and reward (red line), along with the \n  result of applying the same policy to the real environment (black lines). \n  The model's uncertainty is visualized by plotting lines the maximum and \n  minimum predictions at each time step. The model and policy are specified \n  by passing directories containing configuration files for each; they can \n  be trained independently. The following gif shows an example of 200 steps \n  of pre-trained MBPO policy on Inverted Pendulum environment.\n  \\\n  \\\n  ![Example of Visualizer](http://raw.githubusercontent.com/facebookresearch/mbrl-lib/main/docs/resources/inv_pendulum_mbpo_vis.gif)\n  \u003cbr\u003e\n  \u003cbr\u003e\n* ``DatasetEvaluator``: Loads a pre-trained model and a dataset (can be loaded from separate directories), \n  and computes predictions of the model for each output dimension. The evaluator then\n  creates a scatter plot for each dimension comparing the ground truth output \n  vs. the model's prediction. If the model is an ensemble, the plot shows the\n  mean prediction as well as the individual predictions of each ensemble member.\n  \\\n  \\\n  ![Example of DatasetEvaluator](http://raw.githubusercontent.com/facebookresearch/mbrl-lib/main/docs/resources/dataset_evaluator.png)\n  \u003cbr\u003e\n  \u003cbr\u003e\n* ``FineTuner``: Can be used to train a model on a dataset produced by a given agent/controller. \n  The model and agent can be loaded from separate directories, and the fine tuner will roll the \n  environment for some number of steps using actions obtained from the \n  controller. The final model and dataset will then be saved under directory\n  \"model_dir/diagnostics/subdir\", where `subdir` is provided by the user.\\\n  \u003cbr\u003e\n* ``True Dynamics Multi-CPU Controller``: This script can run\na trajectory optimizer agent on the true environment using Python's \n  multiprocessing. Each environment runs in its own CPU, which can significantly\n  speed up costly sampling algorithm such as CEM. The controller will also save\n  a video if the ``render`` argument is passed. Below is an example on \n  HalfCheetah-v2 using CEM for trajectory optimization. To specify the environment,\n  follow the single string syntax described \n  [here](https://github.com/facebookresearch/mbrl-lib/blob/main/README.md#supported-environments).\n  \\\n  \\\n  ![Control Half-Cheetah True Dynamics](http://raw.githubusercontent.com/facebookresearch/mbrl-lib/main/docs/resources/halfcheetah-break.gif)\n  \u003cbr\u003e\n  \u003cbr\u003e\n* [``TrainingBrowser``](training_browser.py): This script launches a lightweight\ntraining browser for plotting rewards obtained after training runs \n  (as long as the runs use our logger). \n  The browser allows aggregating multiple runs and displaying mean/std, \n  and also lets the user save the image to hard drive. The legend and axes labels\n  can be edited in the pane at the bottom left. Requires installing `PyQt5`. \n  Thanks to [a3ahmad](https://github.com/a3ahmad) for the contribution!\n\n  ![Training Browser Example](http://raw.githubusercontent.com/facebookresearch/mbrl-lib/main/docs/resources/training-browser-example.png)\n\nNote that, except for the training browser, all the tools above require Mujoco \ninstallation and are specific to models of type \n[``OneDimTransitionRewardModel``](../models/one_dim_tr_model.py).\nWe are planning to extend this in the future; if you have useful suggestions\ndon't hesitate to raise an issue or submit a pull request!\n\n## Advanced Examples\nMBRL-Lib can be used for many different research projects in the subject area. \nBelow are some community-contributed examples:\n*  [Trajectory-based Dynamics Model](https://arxiv.org/abs/2012.09156) Training [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/natolambert/mbrl-lib-dev/blob/main/notebooks/traj_based_model.ipynb) \n\n* Using HuggingFace Hub Integration for saving and loading models: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1Awp4sSGFgz8nARyfonBGsLrgRWrbqChx?usp=sharing#scrollTo=SX1JYR_bJNWf)\n\n## Documentation \nPlease check out our **[documentation](https://facebookresearch.github.io/mbrl-lib/)** \nand don't hesitate to raise issues or contribute if anything is unclear!\n\n## License\n`mbrl` is released under the MIT license. See [LICENSE](LICENSE) for \nadditional details about it. See also our \n[Terms of Use](https://opensource.facebook.com/legal/terms) and \n[Privacy Policy](https://opensource.facebook.com/legal/privacy).\n\n## Citing\nIf you use this project in your research, please cite:\n\n```BibTeX\n@Article{Pineda2021MBRL,\n  author  = {Luis Pineda and Brandon Amos and Amy Zhang and Nathan O. Lambert and Roberto Calandra},\n  journal = {Arxiv},\n  title   = {MBRL-Lib: A Modular Library for Model-based Reinforcement Learning},\n  year    = {2021},\n  url     = {https://arxiv.org/abs/2104.10159},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffacebookresearch%2Fmbrl-lib","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffacebookresearch%2Fmbrl-lib","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffacebookresearch%2Fmbrl-lib/lists"}