{"id":30957255,"url":"https://github.com/filippoairaldi/mpc-reinforcement-learning","last_synced_at":"2025-10-30T20:57:28.613Z","repository":{"id":64971897,"uuid":"571672052","full_name":"FilippoAiraldi/mpc-reinforcement-learning","owner":"FilippoAiraldi","description":"Reinforcement Learning with Model Predictive Control","archived":false,"fork":false,"pushed_at":"2025-08-21T17:06:32.000Z","size":3918,"stargazers_count":522,"open_issues_count":0,"forks_count":56,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-08-26T11:39:59.339Z","etag":null,"topics":["casadi","model-predictive-control","optimization","reinforcement-learning"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/mpcrl/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FilippoAiraldi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-11-28T16:27:04.000Z","updated_at":"2025-08-25T20:37:27.000Z","dependencies_parsed_at":"2024-01-10T20:50:28.381Z","dependency_job_id":"70eb029d-df39-47fe-a280-869ced7d5433","html_url":"https://github.com/FilippoAiraldi/mpc-reinforcement-learning","commit_stats":{"total_commits":498,"total_committers":3,"mean_commits":166.0,"dds":0.3293172690763052,"last_synced_commit":"c78da07181be5baa88bb724204288711d7cd1d20"},"previous_names":[],"tags_count":64,"template":false,"template_full_name":null,"purl":"pkg:github/FilippoAiraldi/mpc-reinforcement-learning","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FilippoAiraldi%2Fmpc-reinforcement-learning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FilippoAiraldi%2Fmpc-reinforcement-learning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FilippoAiraldi%2Fmpc-reinforcement-learning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FilippoAiraldi%2Fmpc-reinforcement-learning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FilippoAiraldi","download_url":"https://codeload.github.com/FilippoAiraldi/mpc-reinforcement-learning/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FilippoAiraldi%2Fmpc-reinforcement-learning/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274648319,"owners_count":25324299,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-11T02:00:13.660Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["casadi","model-predictive-control","optimization","reinforcement-learning"],"created_at":"2025-09-11T13:45:01.196Z","updated_at":"2025-10-30T20:57:28.589Z","avatar_url":"https://github.com/FilippoAiraldi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Reinforcement Learning with Model Predictive Control\n\n**M**odel **P**redictive **C**ontrol-based **R**einforcement **L**earning (**mpcrl**,\nfor short) is a library for training model-based Reinforcement Learning (RL) [[1]](#1)\nagents with Model Predictive Control (MPC) [[2]](#2) as function approximation.\n\n\u003e |   |   |\n\u003e |---|---|\n\u003e | **Documentation** | \u003chttps://mpc-reinforcement-learning.readthedocs.io/en/stable/\u003e         |\n\u003e | **Download**      | \u003chttps://pypi.python.org/pypi/mpcrl/\u003e                                  |\n\u003e | **Source code**   | \u003chttps://github.com/FilippoAiraldi/mpc-reinforcement-learning/\u003e        |\n\u003e | **Report issues** | \u003chttps://github.com/FilippoAiraldi/mpc-reinforcement-learning/issues/\u003e |\n\n[![PyPI version](https://badge.fury.io/py/mpcrl.svg)](https://badge.fury.io/py/mpcrl)\n[![Source Code License](https://img.shields.io/badge/license-MIT-blueviolet)](https://github.com/FilippoAiraldi/mpc-reinforcement-learning/blob/main/LICENSE)\n![Python 3.9](https://img.shields.io/badge/python-\u003e=3.9-green.svg)\n\n[![Tests](https://github.com/FilippoAiraldi/mpc-reinforcement-learning/actions/workflows/tests.yml/badge.svg)](https://github.com/FilippoAiraldi/mpc-reinforcement-learning/actions/workflows/tests.yml)\n[![Docs](https://readthedocs.org/projects/mpc-reinforcement-learning/badge/?version=stable)](https://mpc-reinforcement-learning.readthedocs.io/en/stable/?badge=stable)\n[![Downloads](https://static.pepy.tech/badge/mpcrl)](https://www.pepy.tech/projects/mpcrl)\n[![Maintainability](https://qlty.sh/gh/FilippoAiraldi/projects/mpc-reinforcement-learning/maintainability.svg)](https://qlty.sh/gh/FilippoAiraldi/projects/mpc-reinforcement-learning)\n[![Code Coverage](https://qlty.sh/gh/FilippoAiraldi/projects/mpc-reinforcement-learning/coverage.svg)](https://qlty.sh/gh/FilippoAiraldi/projects/mpc-reinforcement-learning)\n[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://docs.astral.sh/ruff/)\n\n---\n\n## Introduction\n\nThis framework, also referred to as _RL with/using MPC_, was first proposed in [[3]](#3)\nand has so far been shown effective in various\napplications, with different learning algorithms and more sound theory, e.g., [[4](#4),\n[5](#5), [7](#7), [8](#8)]. It merges two powerful control techinques into a single\ndata-driven one\n\n- MPC, a well-known control methodology that exploits a prediction model to predict the\n  future behaviour of the environment and compute the optimal action\n\n- and RL, a Machine Learning paradigm that showed many successes in recent years (with\n  games such as chess, Go, etc.) and is highly adaptable to unknown and complex-to-model\n  environments.\n\nThe figure below shows the main idea behind this learning-based control approach. The\nMPC controller, parametrized in its objective, predictive model and constraints (or a\nsubset of these), acts both as policy provider (i.e., providing an action to the\nenvironment, given the current state) and as function approximation for the state and\naction value functions (i.e., predicting the expected return following the current\ncontrol policy from the given state and state-action pair). Concurrently, an RL\nalgorithm is employed to tune this parametrization of the MPC in such a way to increase\nthe controller's performance and achieve an (sub)optimal policy. For this purpose,\ndifferent algorithms can be employed, two of the most successful being Q-learning\n[[4]](#4) and Deterministic Policy Gradient (DPG) [[5]](#5).\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/FilippoAiraldi/mpc-reinforcement-learning/main/docs/_static/mpcrl.diagram.light.png\" alt=\"mpcrl-diagram\" height=\"300\"\u003e\n\u003c/div\u003e\n\n---\n\n## Installation\n\n### Using `pip`\n\nYou can use `pip` to install **mpcrl** with the command\n\n```bash\npip install mpcrl\n```\n\n**mpcrl** has the following dependencies\n\n- Python 3.9 or higher (though support and testing for 3.9 are deprecated)\n- [csnlp](https://casadi-nlp.readthedocs.io/en/stable/)\n- [SciPy](https://scipy.org/)\n- [Gymnasium](https://gymnasium.farama.org/)\n- [Numba](https://numba.pydata.org/)\n- [typing_extensions](https://pypi.org/project/typing-extensions/) (only for Python 3.9)\n\nIf you'd like to play around with the source code instead, run\n\n```bash\ngit clone https://github.com/FilippoAiraldi/mpc-reinforcement-learning.git\n```\n\nThe `main` branch contains the main releases of the packages (and the occasional post\nrelease). The `experimental` branch is reserved for the implementation and test of new\nfeatures and hosts the release candidates. You can then install the package to edit it\nas you wish as\n\n```bash\npip install -e /path/to/mpc-reinforcement-learning\n```\n\n---\n\n## Getting started\n\nHere we provide the skeleton of a simple application of the library. The aim of the code\nbelow is to let an MPC control strategy learn how to optimally control a simple Linear\nTime Invariant (LTI) system. The cost (i.e., the opposite of the reward) of controlling\nthis system in state $s \\in \\mathbb{R}^{n_s}$ with action\n$a \\in \\mathbb{R}^{n_a}$ is given by\n\n$$\nL(s,a) = s^\\top Q s + a^\\top R a,\n$$\n\nwhere $Q \\in \\mathbb{R}^{n_s \\times n_s}$ and $R \\in \\mathbb{R}^{n_a \\times n_a}$ are\nsuitable positive definite matrices. This is a very well-known problem in optimal\ncontrol theory. However, here, in the context of RL, these matrices are not known, and\nwe can only observe realizations of the cost for each state-action pair our controller\nvisits. The underlying system dynamics are described by the usual state-space model\n\n$$\ns_{k+1} = A s_k + B a_k,\n$$\n\nwhose matrices $A \\in \\mathbb{R}^{n_s \\times n_s}$ and\n$B \\in \\mathbb{R}^{n_s \\times n_a}$ could again in general be unknown. The control\naction $a_k$ is assumed bounded in the interval $[-1,1]$. In what follows we will go\nthrough the usual steps in setting up and solving such a task.\n\n### Environment\n\nThe first ingredient to implement is the LTI system in the form of a `gymnasium.Env`\nclass. Fill free to fill in the missing parts based on your needs. The\n`gymnasium.Env.reset` method should initialize the state of the system, while the\n`gymnasium.Env.step` method should update the state of the system based on the action\nprovided and mainly return the new state and the cost.\n\n```python\nfrom gymnasium import Env\nfrom gymnasium.wrappers import TimeLimit\nimport numpy as np\n\n\nclass LtiSystem(Env):\n    ns = ...  # number of states (must be continuous)\n    na = ...  # number of actions (must be continuous)\n    A = ...  # state-space matrix A\n    B = ...  # state-space matrix B\n    Q = ...  # state-cost matrix Q\n    R = ...  # action-cost matrix R\n    action_space = Box(-1.0, 1.0, (na,), np.float64)  # action space\n\n    def reset(self, *, seed=None, options=None):\n        super().reset(seed=seed, options=options)\n        self.s = ...  # set initial state\n        return self.s, {}\n\n    def step(self, action):\n        a = np.reshape(action, self.action_space.shape)\n        assert self.action_space.contains(a)\n        c = self.s.T @ self.Q @ self.s + a.T @ self.R @ a\n        self.s = self.A @ self.s + self.B @ a\n        return self.s, c, False, False, {}\n\n\n# lastly, instantiate the environment with a wrapper to ensure the simulation finishes\nenv = TimeLimit(LtiSystem(), max_steps=5000)\n```\n\n### Controller\n\nAs aforementioned, we'd like to control this system via an MPC controller. Therefore,\nthe next step is to craft one. To do so, we leverage the `csnlp` package, in particular\nits `csnlp.wrappers.Mpc` class (on top of that, under the hood, we exploit this package\nalso to compute the sensitivities of the MPC controller w.r.t. its parametrization,\nwhich are crucial in calculating the RL updates). In mathematical terms, the MPC looks\nlike this:\n\n$$\n\\begin{aligned}\n  \\min_{x_{0:N}, u_{0:N-1}} \\quad \u0026 \\sum_{i=0}^{N-1}{ x_i^\\top \\tilde{Q} x_i + u_i^\\top \\tilde{R} u_i } \u0026 \\\\\n  \\textrm{s.t.} \\quad \u0026 x_0 = s_k \\\\\n                      \u0026 x_{i+1} = \\tilde{A} x_i + \\tilde{B} u_i, \\quad \u0026 i=0,\\dots,N-1 \\\\\n                      \u0026 -1 \\le u_k \\le 1, \\quad \u0026 i=0,\\dots,N-1\n\\end{aligned}\n$$\n\nwhere $\\tilde{Q}, \\tilde{R}, \\tilde{A}, \\tilde{B}$ do not necessarily have to match\nthe environment's $Q, R, A, B$ as they represent a possibly approximated a priori\nknowledge on the sytem. In code, we can implement this as follows.\n\n```python\nimport casadi as cs\nfrom csnlp import Nlp\nfrom csnlp.wrappers import Mpc\n\nN = ...  # prediction horizon\nmpc = Mpc[cs.SX](Nlp(), N)\n\n# create the parametrization of the controller\nnx, nu = LtiSystem.ns, LtiSystem.na\nAtilde = mpc.parameter(\"Atilde\", (nx, nx))\nBtilde = mpc.parameter(\"Btilde\", (nx, nu))\nQtilde = mpc.parameter(\"Qtilde\", (nx, nx))\nRtilde = mpc.parameter(\"Rtilde\", (nu, nu))\n\n# create the variables of the controller\nx, _ = mpc.state(\"x\", nx)\nu, _ = mpc.action(\"u\", nu, lb=-1.0, ub=1.0)\n\n# set the dynamics\nmpc.set_linear_dynamics(Atilde, Btilde)\n\n# set the objective\nmpc.minimize(\n    sum(cs.bilin(Qtilde, x[:, i]) + cs.bilin(Rtilde, u[:, i]) for i in range(N))\n)\n\n# initiliaze the solver with some options\nopts = {\n    \"print_time\": False,\n    \"bound_consistency\": True,\n    \"calc_lam_x\": True,\n    \"calc_lam_p\": False,\n    \"ipopt\": {\"max_iter\": 500, \"sb\": \"yes\", \"print_level\": 0},\n}\nmpc.init_solver(opts, solver=\"ipopt\")\n```\n\n### Learning\n\nThe last step is to train the controller using an RL algorithm. For instance, here we\nuse Q-Learning. The idea is to let the controller interact with the environment, observe\nthe cost, and update the MPC parameters accordingly. This can be achieved by computing\nthe temporal difference error\n\n$$\n\\delta_k = L(s_k, a_k) + \\gamma V_\\theta(s_{k+1}) - Q_\\theta(s_k, a_k),\n$$\n\nwhere $\\gamma$ is the discount factor, and $V_\\theta$ and $Q_\\theta$ are the state and\nstate-action value functions, both provided by the parametrized MPC controller with\n$\\theta = \\{\\tilde{A}, \\tilde{B}, \\tilde{Q}, \\tilde{R}\\}$. The update rule for the\nparameters is then given by\n\n$$\n\\theta \\gets \\theta + \\alpha \\delta_k \\nabla_\\theta Q_\\theta(s_k, a_k),\n$$\n\nwhere $\\alpha$ is the learning rate, and $\\nabla_\\theta Q_\\theta(s_k, a_k)$ is the\nsensitivity of the state-action value function w.r.t. the parameters. All of this can be\nimplemented as follows.\n\n```python\nfrom mpcrl import LearnableParameter, LearnableParametersDict, LstdQLearningAgent\nfrom mpcrl.optim import GradientDescent\n\n# give some initial values to the learnable parameters (shapes must match!)\nlearnable_pars_init = {\"Atilde\": ..., \"Btilde\": ..., \"Qtilde\": ..., \"Rtilde\": ...}\n\n# create the set of parameters that should be learnt\nlearnable_pars = LearnableParametersDict(\n    (\n        LearnableParameter(name, val.shape, val)\n        for name, val in learnable_pars_init.items()\n    )\n)\n\n# instantiate the learning agent\nagent = LstdQLearningAgent(\n    mpc=mpc,\n    learnable_parameters=learnable_pars,\n    discount_factor=...,  # a number in (0,1], e.g.,  1.0\n    update_strategy=...,  # an integer, e.g., 1\n    optimizer=GradientDescent(learning_rate=...),\n    record_td_errors=True,\n)\n\n# finally, launch the training for 5000 timesteps. The method will return an array of\n# (hopefully) decreasing costs\ncosts = agent.train(env=env, episodes=1, seed=69)\n```\n\n---\n\n## Examples\n\nOur\n[examples](https://mpc-reinforcement-learning.readthedocs.io/en/stable/auto_examples/index.html)\nsubdirectory contains examples on how to use the library on some academic, small-scale\napplication (a small linear time-invariant (LTI) system), tackled both with\n[on-policy Q-learning](https://mpc-reinforcement-learning.readthedocs.io/en/stable/auto_examples/gradient-based-onpolicy/q_learning.html#sphx-glr-auto-examples-gradient-based-onpolicy-q-learning-py),\n[off-policy Q-learning](https://mpc-reinforcement-learning.readthedocs.io/en/stable/auto_examples/gradient-based-offpolicy/q_learning_offpolicy.html#sphx-glr-auto-examples-gradient-based-offpolicy-q-learning-offpolicy-py)\nand\n[DPG](https://mpc-reinforcement-learning.readthedocs.io/en/stable/auto_examples/gradient-based-onpolicy/dpg.html#sphx-glr-auto-examples-gradient-based-onpolicy-dpg-py).\nWhile the aforementioned algorithms are all gradient-based, we also provide an\n[example on how to use Bayesian Optimization (BO)](https://mpc-reinforcement-learning.readthedocs.io/en/stable/auto_examples/gradient-free/bayesopt.html#sphx-glr-auto-examples-gradient-free-bayesopt-py)\n[[6]](#6) to tune the MPC parameters in a gradient-free way.\n\n---\n\n## License\n\nThe repository is provided under the MIT License. See the LICENSE file included with\nthis repository.\n\n---\n\n## Author\n\n[Filippo Airaldi](https://www.tudelft.nl/staff/f.airaldi/), PhD Candidate\n[f.airaldi@tudelft.nl | filippoairaldi@gmail.com]\n\n\u003e [Delft Center for Systems and Control](https://www.tudelft.nl/en/me/about/departments/delft-center-for-systems-and-control/)\nin [Delft University of Technology](https://www.tudelft.nl/en/)\n\nCopyright (c) 2025 Filippo Airaldi.\n\nCopyright notice: Technische Universiteit Delft hereby disclaims all copyright interest\nin the program “mpcrl” (Reinforcement Learning with Model Predictive Control) written by\nthe Author(s). Prof. Dr. Ir. Fred van Keulen, Dean of ME.\n\n---\n\n## References\n\n\u003ca id=\"1\"\u003e[1]\u003c/a\u003e\nSutton, R.S. and Barto, A.G. (2018).\n[Reinforcement learning: An introduction](https://mitpress-mit-edu.tudelft.idm.oclc.org/9780262039246/reinforcement-learning/).\nCambridge, MIT press.\n\n\u003ca id=\"2\"\u003e[2]\u003c/a\u003e\nRawlings, J.B., Mayne, D.Q. and Diehl, M. (2017).\n[Model Predictive Control: theory, computation, and design (Vol. 2)](https://sites.engineering.ucsb.edu/~jbraw/mpc/).\nMadison, WI: Nob Hill Publishing.\n\n\u003ca id=\"3\"\u003e[3]\u003c/a\u003e\nGros, S. and Zanon, M. (2020).\n[Data-Driven Economic NMPC Using Reinforcement Learning](https://ieeexplore-ieee-org.tudelft.idm.oclc.org/document/8701462).\nIEEE Transactions on Automatic Control, 65(2), 636-648.\n\n\u003ca id=\"4\"\u003e[4]\u003c/a\u003e\nEsfahani, H. N. and Kordabad,  A. B. and Gros, S. (2021).\n[Approximate Robust NMPC using Reinforcement Learning](https://ieeexplore-ieee-org.tudelft.idm.oclc.org/document/9655129).\nEuropean Control Conference (ECC), 132-137.\n\n\u003ca id=\"5\"\u003e[5]\u003c/a\u003e\nCai, W. and Kordabad, A. B. and Esfahani, H. N. and Lekkas, A. M. and Gros, S. (2021).\n[MPC-based Reinforcement Learning for a Simplified Freight Mission of Autonomous Surface Vehicles](https://ieeexplore-ieee-org.tudelft.idm.oclc.org/document/9683750).\n60th IEEE Conference on Decision and Control (CDC), 2990-2995.\n\n\u003ca id=\"6\"\u003e[6]\u003c/a\u003e\nGarnett, R., 2023. [Bayesian Optimization](https://bayesoptbook.com/).\nCambridge University Press.\n\n\u003ca id=\"7\"\u003e[7]\u003c/a\u003e\nGros, S. and Zanon, M. (2022).\n[Learning for MPC with stability \u0026 safety guarantees](https://www.sciencedirect.com/science/article/pii/S0005109822004605).\nAutomatica, 164, 110598.\n\n\u003ca id=\"8\"\u003e[8]\u003c/a\u003e\nZanon, M. and Gros, S. (2021).\n[Safe Reinforcement Learning Using Robust MPC](https://ieeexplore.ieee.org/abstract/document/9198135/).\nIEEE Transactions on Automatic Control, 66(8), 3638-3652.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffilippoairaldi%2Fmpc-reinforcement-learning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffilippoairaldi%2Fmpc-reinforcement-learning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffilippoairaldi%2Fmpc-reinforcement-learning/lists"}