{"id":22321754,"url":"https://github.com/chauncygu/multi-agent-constrained-policy-optimisation","last_synced_at":"2026-03-04T16:31:00.837Z","repository":{"id":40437262,"uuid":"414188721","full_name":"chauncygu/Multi-Agent-Constrained-Policy-Optimisation","owner":"chauncygu","description":"Multi-Agent Constrained Policy Optimisation (MACPO; MAPPO-L).","archived":false,"fork":false,"pushed_at":"2024-04-17T02:25:35.000Z","size":8895,"stargazers_count":181,"open_issues_count":4,"forks_count":29,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-07-29T15:08:28.035Z","etag":null,"topics":["multi-agent-reinforcement-learning","policy-optimization","safe-reinforcement-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chauncygu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-06T11:43:43.000Z","updated_at":"2025-07-22T02:28:25.000Z","dependencies_parsed_at":"2024-12-04T00:34:01.975Z","dependency_job_id":null,"html_url":"https://github.com/chauncygu/Multi-Agent-Constrained-Policy-Optimisation","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/chauncygu/Multi-Agent-Constrained-Policy-Optimisation","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chauncygu%2FMulti-Agent-Constrained-Policy-Optimisation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chauncygu%2FMulti-Agent-Constrained-Policy-Optimisation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chauncygu%2FMulti-Agent-Constrained-Policy-Optimisation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chauncygu%2FMulti-Agent-Constrained-Policy-Optimisation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chauncygu","download_url":"https://codeload.github.com/chauncygu/Multi-Agent-Constrained-Policy-Optimisation/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chauncygu%2FMulti-Agent-Constrained-Policy-Optimisation/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30086451,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-04T15:40:14.053Z","status":"ssl_error","status_checked_at":"2026-03-04T15:40:13.655Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["multi-agent-reinforcement-learning","policy-optimization","safe-reinforcement-learning"],"created_at":"2024-12-04T00:22:44.311Z","updated_at":"2026-03-04T16:31:00.808Z","avatar_url":"https://github.com/chauncygu.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Multi-Agent Constrained Policy Optimisation (MACPO)\r\n\r\nThe repository is for the paper: **[Multi-Agent Constrained Policy Optimisation](http://arxiv.org/abs/2110.02793)**, in which we investigate the problem of safe MARL. The problem of safe multi-agent learning with safety constraints has not been rigorously studied; very few solutions have been proposed, nor a sharable testing environment or benchmarks.   To fill these gaps, in this work, we formulate the safe multi-agent reinforcement learning problem as a constrained Markov game and solve it with trust region methods. Our solutions---*Multi-Agent Constrained Policy Optimisation (MACPO)* and *MAPPO-Lagrangian*---leverage on the theory of  *Constrained Policy Optimisation (CPO)* and multi-agent trust region learning, and critically, they enjoy theoretical guarantees of  both  monotonic improvement in reward and satisfaction of safety constraints  at every iteration. Experimental results reveal that  *MACPO/MAPPO-Lagrangian* significantly outperform baselines in terms of balancing the performance and constraint satisfaction, e.g. [MAPPO](https://arxiv.org/abs/2103.01955), [IPPO](https://arxiv.org/abs/2011.09533), [HAPPO](https://arxiv.org/abs/2109.11251).\r\n\r\n\r\n\r\n## Environments Supported:\r\n\r\n- [Safety Multi-Agent Mujoco](https://github.com/chauncygu/Safe-Multi-Agent-Mujoco)\r\n\r\n\r\n\r\n\r\n## 1. Installation\r\n\r\n####  1.1 Create Environment\r\n\r\n``` Bash\r\n# create conda environment\r\nconda create -n macpo python==3.7\r\nconda activate macpo\r\npip install -r requirements.txt\r\nconda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia\r\n```\r\n\r\n```\r\ncd MACPO/macpo (for the macpo algorithm) or cd MAPPO-Lagrangian/mappo_lagrangian (for the mappo_lagrangian algorithm)\r\npip install -e .\r\n```\r\n\r\n\r\n\r\n#### 1.2 Install Safety Multi-Agent Mujoco\r\n\r\n\r\n- Install mujoco accoring to [mujoco-py](https://github.com/openai/mujoco-py) and [MuJoCo website](https://www.roboti.us/license.html).\r\n- clone [Safety Multi-Agent Mujoco](https://github.com/chauncygu/Safe-Multi-Agent-Mujoco) to the env path (in this repository, have set the path).\r\n\r\n``` Bash\r\nLD_LIBRARY_PATH=${HOME}/.mujoco/mujoco200/bin;\r\nLD_PRELOAD=/usr/lib/x86_64-linux-gnu/libGLEW.so\r\n```\r\n\r\n\r\n\r\n## 2. Train\r\n\r\n```\r\ncd MACPO/macpo/scripts or cd MAPPO-Lagrangian/mappo_lagrangian/scripts\r\nchmod +x ./train_mujoco.sh\r\n./train_mujoco.sh\r\n```\r\n\r\n\r\n## 3. Results\r\n\r\n\u003cdiv align=center\u003e\r\n\u003cimg src=\"https://github.com/chauncygu/Multi-Agent-Constrained-Policy-Optimisation/blob/main/figures/wall1_manyagent_ant.png\" width=\"850\"/\u003e    \r\n\u003cimg src=\"https://github.com/chauncygu/Multi-Agent-Constrained-Policy-Optimisation/blob/main/figures/New_Ant_results.png\" width=\"850\"/\u003e \r\n\u003cimg src=\"https://github.com/chauncygu/Multi-Agent-Constrained-Policy-Optimisation/blob/main/figures/New_HalfCheetah_results.png\" width=\"850\"/\u003e\r\n\u003c/div\u003e\r\n    \r\n\u003cdiv align=center\u003e\r\n\u003ccenter style=\"color:#000000;text-decoration:underline\"\u003e\r\n    Performance comparisons on tasks of Safe ManyAgent Ant, Safe Ant, and Safe HalfCheetah in terms of cost (the first row) and reward (the second row). The  safety constraint  values are: 1 for ManyAgent Ant, 0.2 for Ant, and 5 for HalfCheetah. Our methods consistently achieve almost zero costs, thus satisfying safe constraints,  on all tasks. In terms of reward, our methods outperform \u003ca href=\"https://arxiv.org/abs/2011.09533)\"\u003eIPPO\u003c/a\u003e and \u003ca href=\"https://arxiv.org/abs/2103.01955\"\u003eMAPPO\u003c/a\u003e on some tasks  but  underperform  \u003ca href=\"https://arxiv.org/abs/2109.11251\"\u003eHAPPO\u003c/a\u003e, which  is also an unsafe algorithm.\u003c/center\u003e\r\n\u003c/div\u003e\r\n\r\n\r\n## 4. Demos\r\n\r\n\u003c!--\u003cdiv align=center\u003e\r\n\u003cimg src=\"https://github.com/chauncygu/Multi-Agent-Constrained-Policy-Optimisation/blob/main/figures/202111071600-unsafe-end%2000_00_00-00_00_30.gif\" width=\"700\"/\u003e    \r\n\u003c/div\u003e\r\n    \r\n\u003cdiv align=center\u003e\r\n\u003ccenter style=\"color:#000000;text-decoration:underline\"\u003e\r\n    A demo denotes \u003cb\u003eunsafe\u003c/b\u003e performance using \u003ca href=\"https://arxiv.org/abs/2109.11251\"\u003eHAPPO\u003c/a\u003e on Ant2x4 task.\u003c/center\u003e\r\n\u003c/div\u003e\r\n\r\n\u0026nbsp;\r\n\u003cdiv align=center\u003e\r\n\u003cimg src=\"https://github.com/chauncygu/Multi-Agent-Constrained-Policy-Optimisation/blob/main/figures/202111140948-safe-end1%2000_00_00-00_00_30.gif\" width=\"700\"/\u003e    \r\n\u003c/div\u003e\r\n    \r\n\u003cdiv align=center\u003e\r\n\u003ccenter style=\"color:#000000;text-decoration:underline\"\u003e\r\n    A demo denotes \u003cb\u003esafe\u003c/b\u003e performance using \u003ca href=\"http://arxiv.org/abs/2110.02793\"\u003eMAPPO-Lagrangian\u003c/a\u003e on Ant2x4 task.\u003c/center\u003e\r\n\u003c/div\u003e--\u003e\r\n\r\n**Ant Task**: the width of the corridor set by two walls is 10 m. The environment emits the cost of 1 for an agent, if the distance between the robot and the wall is less than 1.8 m, or when the robot topples over.\r\n\r\n|![](https://github.com/chauncygu/Multi-Agent-Constrained-Policy-Optimisation/blob/main/figures/202111071600-unsafe-end%2000_00_00-00_00_30.gif)|![](https://github.com/chauncygu/Multi-Agent-Constrained-Policy-Optimisation/blob/main/figures/202111140948-safe-end1%2000_00_00-00_00_30.gif)|\r\n| :---: | :---: | \r\n|  A demo denotes \u003cb\u003eunsafe\u003c/b\u003e performance using \u003ca href=\"https://arxiv.org/abs/2109.11251\"\u003eHAPPO\u003c/a\u003e on Ant-2x4 task.\u003c/center\u003e | A demo denotes \u003cb\u003esafe\u003c/b\u003e performance using \u003ca href=\"http://arxiv.org/abs/2110.02793\"\u003eMAPPO-Lagrangian\u003c/a\u003e on Ant-2x4 task.\u003c/center\u003e | \r\n\r\n\r\n**HalfCheetah Task**: In the task, the agents move inside a corridor (which constraints their movement, but does not induce costs). Together with them, there are bombs moving inside the corridor. If an agent finds itself too close to the bomb, the distance between an agent and the bomb is less than 9m, a cost of 1 will be emitted, at the same time, the bomb will turn blood red.\r\n\r\n|![](https://github.com/chauncygu/Multi-Agent-Constrained-Policy-Optimisation/blob/main/figures/202111140948-halfcheetah-unsafe-end%2000_00_00-00_00_30.gif)|![](https://github.com/chauncygu/Multi-Agent-Constrained-Policy-Optimisation/blob/main/figures/202111140948-halfcheetah-safe-end%2000_00_00-00_00_30.gif)|\r\n| :---: | :---: | \r\n|  A demo denotes \u003cb\u003eunsafe\u003c/b\u003e performance using \u003ca href=\"https://arxiv.org/abs/2109.11251\"\u003eHAPPO\u003c/a\u003e on HalfCheetah-2x3 task.\u003c/center\u003e | A demo denotes \u003cb\u003esafe\u003c/b\u003e performance using \u003ca href=\"http://arxiv.org/abs/2110.02793\"\u003eMAPPO-Lagrangian\u003c/a\u003e on HalfCheetah-2x3 task.\u003c/center\u003e | \r\n\r\n\r\n**ManyAgent Ant Task One**: In the ManyAgent Ant task, the width of the corridor set by two walls is 9m. The environment emits the cost of 1 for an agent, if the distance between the robot and the wall is less than 1.8 m, or when the robot topples over. \r\n\r\n|![](https://github.com/chauncygu/Multi-Agent-Constrained-Policy-Optimisation/blob/main/figures/sadppo-manyagent-ant--unsafe-end-have-word%2000_00_00-00_00_30.gif)|![](https://github.com/chauncygu/Multi-Agent-Constrained-Policy-Optimisation/blob/main/figures/safe-mappo-manyagent-ant--safe-end-have-word-01%2000_00_00-00_00_30.gif)|\r\n| :---: | :---: | \r\n|  A demo denotes \u003cb\u003eunsafe\u003c/b\u003e performance using \u003ca href=\"https://arxiv.org/abs/2109.11251\"\u003eHAPPO\u003c/a\u003e on ManyAgent Ant-2x3 task.\u003c/center\u003e | A demo denotes \u003cb\u003esafe\u003c/b\u003e performance using \u003ca href=\"http://arxiv.org/abs/2110.02793\"\u003eMAPPO-Lagrangian\u003c/a\u003e on ManyAgent Ant-2x3 task.\u003c/center\u003e | \r\n\r\n\r\n**ManyAgent Ant Task Two**: In the ManyAgent Ant task, the width of the corridor is 12 m; its walls fold at the angle of 30 degrees. The environment emits the cost of 1 for an agent, if the distance between the robot and the wall is less than 1.8 m, or when the robot topples over.\r\n\r\n|![](https://github.com/chauncygu/Multi-Agent-Constrained-Policy-Optimisation/blob/main/figures/two-wall-sadppo-unsafe-end-have-word-manyagent-ant%2000_00_00-00_00_30.gif)|![](https://github.com/chauncygu/Multi-Agent-Constrained-Policy-Optimisation/blob/main/figures/two-wall-safe-mappo--manyagent-ant-safe-end-have-word-manyagent-ant%2000_00_00-00_00_30.gif)|![](https://github.com/chauncygu/Multi-Agent-Constrained-Policy-Optimisation/blob/main/figures/two-wall-macpo-manyagent-ant--safe-end-have-word%2000_00_00-00_00_30.gif)|\r\n| :---: | :---: | :---: | \r\n|  A demo denotes \u003cb\u003eunsafe\u003c/b\u003e performance using \u003ca href=\"https://arxiv.org/abs/2109.11251\"\u003eHAPPO\u003c/a\u003e on ManyAgent Ant-2x3 task.\u003c/center\u003e | A demo denotes \u003cb\u003eunsafe\u003c/b\u003e performance using \u003ca href=\"http://arxiv.org/abs/2110.02793\"\u003eMAPPO-Lagrangian\u003c/a\u003e on ManyAgent Ant-2x3 task.\u003c/center\u003e | A demo denotes \u003cb\u003esafe\u003c/b\u003e performance using \u003ca href=\"http://arxiv.org/abs/2110.02793\"\u003eMACPO\u003c/a\u003e on ManyAgent Ant-2x3 task.\u003c/center\u003e | \r\n\r\n\r\n## 5. Publication\r\nIf you find the repository useful, please cite the [paper](https://arxiv.org/abs/2110.02793):\r\n```\r\n@article{gu2023safe,\r\n  title={Safe Multi-Agent Reinforcement Learning for Multi-Robot Control},\r\n  author={Gu, Shangding and Kuba, Jakub Grudzien and Chen, Yuanpei and Du, Yali and Yang, Long and Knoll, Alois and Yang, Yaodong},\r\n  journal={Artificial Intelligence},\r\n  pages={103905},\r\n  year={2023},\r\n  publisher={Elsevier}\r\n}\r\n\r\n\r\n```\r\n\r\n\r\n## Acknowledgments\r\n\r\nWe thank the list of contributors from the following open source repositories: [MAPPO](https://github.com/marlbenchmark/on-policy), [HAPPO](https://github.com/cyanrain7/Trust-Region-Policy-Optimisation-in-Multi-Agent-Reinforcement-Learning), [safety-starter-agents](https://github.com/openai/safety-starter-agents), [CMBPO](https://github.com/anyboby/Constrained-Model-Based-Policy-Optimization).\r\n\r\n\r\n\r\n\r\n\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchauncygu%2Fmulti-agent-constrained-policy-optimisation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchauncygu%2Fmulti-agent-constrained-policy-optimisation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchauncygu%2Fmulti-agent-constrained-policy-optimisation/lists"}