{"id":16394532,"url":"https://github.com/stweigand97/model-based-pde-control","last_synced_at":"2026-01-19T03:07:38.726Z","repository":{"id":246380179,"uuid":"610939815","full_name":"stwerner97/model-based-pde-control","owner":"stwerner97","description":"Numerical Evidence for Sample Efficiency of Model-Based over Model-Free Reinforcement Learning Control of Partial Differential Equations [ECC'24]","archived":false,"fork":false,"pushed_at":"2024-07-31T20:27:00.000Z","size":27422,"stargazers_count":4,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-18T17:57:26.355Z","etag":null,"topics":["gym","kuramoto-sivashinsky","mbpo","model-based-reinforcement-learning","partial-differential-equations","proximal-policy-optimization","reinforcement-learning","soft-actor-critic","stable-baselines3","surrogate-models"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stwerner97.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-07T19:38:12.000Z","updated_at":"2025-02-28T08:45:56.000Z","dependencies_parsed_at":null,"dependency_job_id":"a70d580c-1a6a-43e1-b375-755a215a3853","html_url":"https://github.com/stwerner97/model-based-pde-control","commit_stats":null,"previous_names":["stwerner97/pdecontrol"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stwerner97%2Fmodel-based-pde-control","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stwerner97%2Fmodel-based-pde-control/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stwerner97%2Fmodel-based-pde-control/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stwerner97%2Fmodel-based-pde-control/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stwerner97","download_url":"https://codeload.github.com/stwerner97/model-based-pde-control/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245056889,"owners_count":20553855,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gym","kuramoto-sivashinsky","mbpo","model-based-reinforcement-learning","partial-differential-equations","proximal-policy-optimization","reinforcement-learning","soft-actor-critic","stable-baselines3","surrogate-models"],"created_at":"2024-10-11T04:55:56.611Z","updated_at":"2026-01-19T03:07:38.699Z","avatar_url":"https://github.com/stwerner97.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Numerical Evidence for Sample Efficiency of Model-Based over Model-Free Reinforcement Learning Control of Partial Differential Equations [ECC'24]\n[![Python application](https://github.com/stwerner97/pdecontrol/actions/workflows/python-app.yml/badge.svg)](https://github.com/stwerner97/pdecontrol/actions/workflows/python-app.yml) [![Code Style: Black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n\nThis repository holds the implementation of our model-based deep reinforcement learning algorithm for the control of partial differential equations (specifically the Kuramoto-Sivashinsky \u0026 Burger equations). This work has been presented at the European Control Conference\n\n## Controlling the Kuramoto Sivashinsky Equation\n\nOur work aims to minimize the dissipation and power consumption of a system whose dynamics are governed by the Kuramoto-Sivashinsky equation. An example of the system and its rewards where a controller applies random actions to it is illustrated below.\n\n![ks-equation](assets/kuramoto-sivashinsky.gif)\n\nThe reward is an integral of the system's dissipation and power consumption throughout the duration of a control period (fixed time intervals) in which the actuation signals are constant. The controller manipulates the scaling coefficients of Gaussian jets located in the one-dimensional spatial domain.\n\n![control-definition](assets/control-problem.png)\n\n\n## Results \u0026 Evaluation\n\nAfter 50,000 steps, our model-based control algorithm learns a control strategy that is on par with the most competitive model-free strategy after observing ~10x the number of samples, although the soft actor-critic agent eventually converges to a soution that better damps the dissipation of the system.\n\n![mbrl-vs-mfrl](assets/control-results.png)\n\nBoth our model-based agent as well as the model-free soft actor-critic baseline are able to stabilize the system out of the chaotic transient.\n\n![learned-controls](assets/control-policy.png)\n\n## Setup\nThe dependencies can either be installed using the provided ``Dockerfile`` or using the ``pyproject.toml`` file (i.e. ``poetry install .``). Our implementation logs intermediate evaluation results to ``Weights \u0026 Biases``, so an API key must be provided to execute the optimization script.\n\n## Reproduce Our Results\nThe ``pdecontrol/mbrl/script.py`` file serves as an entrypoint to running experiments for our control algorithm. You can use the script to 1) specify the configuration of the agent to be used, 2) adjust its training strategy \u0026 curriculum, and 3) specify the control problem it is applied to.\n\n````python\npython pdecontrol/mbrl/script.py\n    --project \u003cyour wandb project\u003e\n    --name \u003cyour wandb run\u003e\n    --env_id KuramotoSivashinskyEnv-v0\n    --factory \u003cname of dynamics model registered in pdecontrol/architectures/__init__.py\u003e\n    --training \"{\\\"tau\\\": 5, \\\"initial\\\": {\\\"tbtt\\\": 10, \\\"patience\\\": 10, \\\"batch_size\\\": 64}, \\\"iterations\\\": {\\\"tbtt\\\": 10, \\\"patience\\\": 5, \\\"batch_size\\\": 64}}\"\n    --trainer \"{\\\"initial\\\": {\\\"min_steps\\\":250, \\\"max_steps\\\": 2000}, \\\"iterations\\\": {\\\"min_steps\\\":50, \\\"max_steps\\\": 250}}\"\n    --curriculum \"{\\\"scheduler\\\": \\\"LinearScheduler\\\", \\\"steptype\\\": \\\"iteration\\\", \\\"start\\\": 0, \\\"stop\\\": 10, \\\"vmin\\\": 15, \\\"vmax\\\": 15}\"\n    --loss MSELoss\n    --learning_starts 5000\n    --rollout_length_schedule \"{\\\"scheduler\\\": \\\"LinearScheduler\\\", \\\"steptype\\\": \\\"iteration\\\", \\\"start\\\": 0, \\\"stop\\\": 200, \\\"vmin\\\": 3, \\\"vmax\\\": 7}\" --policy_train_steps_per_sample 10\n    --surrogate_train_freq 500\n````\n\n\n# Citation\n\nIf you our code for ``model-based-pde-control`` in your work, please cite our paper:\n\n`````\n@inproceedings{werner2024numerical,\n  title={Numerical Evidence for Sample Efficiency of Model-Based Over Model-Free Reinforcement Learning Control of Partial Differential Equations},\n  author={Werner, Stefan and Peitz, Sebastian},\n  booktitle={2024 European Control Conference (ECC)},\n  pages={2965--2971},\n  year={2024},\n  organization={IEEE}\n}\n`````","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstweigand97%2Fmodel-based-pde-control","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstweigand97%2Fmodel-based-pde-control","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstweigand97%2Fmodel-based-pde-control/lists"}