{"id":20573577,"url":"https://github.com/timf34/ssd","last_synced_at":"2026-04-21T01:01:29.780Z","repository":{"id":43140230,"uuid":"504095837","full_name":"timf34/SSD","owner":"timf34","description":null,"archived":false,"fork":false,"pushed_at":"2022-08-19T13:36:13.000Z","size":2276,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-06T10:32:02.169Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/timf34.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-06-16T09:37:46.000Z","updated_at":"2022-06-16T09:41:27.000Z","dependencies_parsed_at":"2022-08-12T10:20:51.048Z","dependency_job_id":null,"html_url":"https://github.com/timf34/SSD","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/timf34/SSD","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timf34%2FSSD","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timf34%2FSSD/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timf34%2FSSD/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timf34%2FSSD/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/timf34","download_url":"https://codeload.github.com/timf34/SSD/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timf34%2FSSD/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32072323,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-20T21:26:33.338Z","status":"ssl_error","status_checked_at":"2026-04-20T21:26:22.081Z","response_time":94,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T05:28:11.434Z","updated_at":"2026-04-21T01:01:29.736Z","avatar_url":"https://github.com/timf34.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## My own notes:\n\n**Setting up:**\n\nNeed to open up a WSL2 terminal\n\nThen need to run `. venv/bin/activate` to activate the virtual environment \n\n\n\n\n\n[![Build Status](https://travis-ci.com/eugenevinitsky/sequential_social_dilemma_games.svg?branch=master)](https://travis-ci.com/eugenevinitsky/sequential_social_dilemma_games)\n\n# Sequential Social Dilemma Games\nThis repo is an open-source implementation of DeepMind's Sequential Social Dilemma (SSD) multi-agent game-theoretic environments [[1]](https://arxiv.org/abs/1702.03037). SSDs can be thought of as analogous to spatially and temporally extended Prisoner's Dilemma-like games. The reward structure poses a dilemma because individual short-term optimal strategies lead to poor long-term outcomes for the group.\n\nThe implemented environments are structured to be compatible with [OpenAIs gym environments](https://github.com/openai/gym) as well as [RLlib's Multiagent Environment](https://github.com/ray-project/ray/blob/master/rllib/env/multi_agent_env.py)\n\n## Implemented Games\n\n* **Cleanup**: A public goods dilemma in which agents get a reward for consuming apples, but must use a cleaning beam to clean a river in order for apples to grow. While an agent is cleaning the river, other agents can exploit it by consuming the apples that appear.\n\n\u003cimg src=\"images/cleanup.png\" alt=\"Image of the cleanup game\" width=\"170\" height=\"246\"/\u003e\n\n* **Harvest**: A tragedy-of-the-commons dilemma in which apples regrow at a rate that depends on the amount of nearby apples. If individual agents employ an exploitative strategy by greedily consuming too many apples, the collective reward of all agents is reduced.\n\n\u003cimg src=\"images/harvest.png\" alt=\"Image of the Harvest game\" width=\"483\" height=\"187\"/\u003e\n\n\u003cimg src=\"images/schelling.png\" alt=\"Schelling diagrams for Harvest and Cleanup\" width=\"953\" height=\"352\"/\u003e\n\nThe above plot shows the empirical Schelling diagrams for both Cleanup (A) and Harvest (B) (from [[2]](https://arxiv.org/abs/1803.08884)). These diagrams show the payoff that an individual agent can expect if it follows a defecting/exploitative strategy (red) vs a cooperative strategy (blue), given the number of other agents that are cooperating.  We can see that an individual agent can almost always greedily benefit from detecting, but the more agents that defect, the worse the outcomes for all agents.  \n\n# Setup instructions\nTo install the SSD environments:\n### Anaconda/miniconda\n```\ngit clone -b master https://github.com/eugenevinitsky/sequential_social_dilemma_games\ncd sequential_social_dilemma_games\nconda create -n ssd python==3.8.10 # Create a conda virtual environment\n# Patch ray due to https://github.com/ray-project/ray/issues/7946\n# And https://github.com/ray-project/ray/pull/8491\n. conda_uint8_patch.sh\n```\n###\n```\ngit clone -b master https://github.com/eugenevinitsky/sequential_social_dilemma_games\ncd sequential_social_dilemma_games\npython3 -m venv venv # Create a Python virtual environment\n. venv/bin/activate\npip3 install --upgrade pip setuptools wheel\npython3 setup.py develop\npip3 install -r requirements.txt\n# Patch ray due to https://github.com/ray-project/ray/issues/7946\n# And https://github.com/ray-project/ray/pull/8491\n. venv_uint8_patch.sh\n```\n\nTo install sb3|rllib|all requirements for learning:\n```\npip install social-dilemmas[sb3|rllib|all]\n```\n\nIf using RLlib:\n```\n\n```\n\nAfter the setup, you can run experiments like so:\n- To train with default parameters (baseline model cleanup with 2 agents):\n`python3 run_scripts/train.py`\n\n- To train the MOA with 5 agents:\n`python3 run_scripts/train.py --model moa --num_agents 5`\n\nMany more options are available which can be found in [default_args.py](config/default_args.py). A collection of preconfigured training scripts can be found in [run_scripts](run_scripts). \n\nNote that the RLlib initialization time can be rather high (up to 5 minutes) the more agents you use, and the more complex your used model is.\n\n- To train using [Stable-Baselines3](https://github.com/DLR-RM/stable-baselines3) and parameter shared PPO:\n`python3 run_scripts/sb3_train.py --env harvest --num_agents 5`\n\n- To train using [MARL-Baselines3](https://github.com/Rohan138/marl-baselines3) and independent PPO:\n`python3 run_scripts/sb3_independent.py --env harvest --num_agents 5`\n\n- To train using [MARL-Baselines3](https://github.com/Rohan138/marl-baselines3) and independent PPO with inequity aversion:\n- \n`python3 run_scripts/sb3_independent.py --env harvest --num_agents 5 --inequity-averse-reward=True --alpha=5.0 --beta=0.05`\n\n# CUDA, cuDNN and tensorflow-gpu\n\nIf you run into any cuda errors, make sure you've got a [compatible set](https://www.tensorflow.org/install/source#tested_build_configurations) of cuda/cudnn/tensorflow versions installed. However, beware of the following:\n\u003eThe compatibility table given in the tensorflow site does not contain specific minor versions for cuda and cuDNN. However, if the specific versions are not met, there will be an error when you try to use tensorflow. [source](https://stackoverflow.com/a/53727997)\n\n# Tests\nTests are located in the test folder and can be run individually or run by running `python -m pytest`. Many of the less obviously defined rules for the games can be understood by reading the tests, each of which outline some aspect of the game. \n\n# Constructing new environments\nEvery environment that subclasses MapEnv probably needs to implement the following methods\n\n```\n    def custom_reset(self):\n        \"\"\"Reset custom elements of the map. For example, spawn apples\"\"\"\n        pass\n\n    def custom_action(self, agent, action):\n        \"\"\"Execute any custom, non-move actions that may be defined, like fire or clean\"\"\"\n        pass\n\n    def custom_map_update(self):\n        \"\"\"Custom map updates that don't have to do with agent actions\"\"\"\n        pass\n\n    def setup_agents(self):\n        \"\"\"Construct all the agents for the environment\"\"\"\n        raise NotImplementedError\n```\n\n## PPO Results\n\nThe below graphs display results for cleanup/harvest using un-tuned PPO in RLlib. As of yet, A3C remains untested.\n\n**Collective cleanup reward**:\n\n\u003cimg src=\"images/cleanup_collective_reward.svg\" alt=\"Collective reward plot of cleanup\" width=\"460.8\" height=\"345.6\"/\u003e\n\n**Collective harvest reward**:\n\n\u003cimg src=\"images/harvest_collective_reward.svg\" alt=\"Collective reward plot of harvest\" width=\"460.8\" height=\"345.6\"/\u003e\n\n\n## Relevant papers\n\n1. Leibo, J. Z., Zambaldi, V., Lanctot, M., Marecki, J., \u0026 Graepel, T. (2017). [Multi-agent reinforcement learning in sequential social dilemmas](https://arxiv.org/abs/1702.03037). In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems (pp. 464-473).\n\n2.  Hughes, E., Leibo, J. Z., Phillips, M., Tuyls, K., Dueñez-Guzman, E., Castañeda, A. G., Dunning, I., Zhu, T., McKee, K., Koster, R., Tina Zhu, Roff, H., Graepel, T. (2018). [Inequity aversion improves cooperation in intertemporal social dilemmas](https://arxiv.org/abs/1803.08884). In Advances in Neural Information Processing Systems (pp. 3330-3340).\n\n3. Jaques, N., Lazaridou, A., Hughes, E., Gulcehre, C., Ortega, P. A., Strouse, D. J., Leibo, J. Z. \u0026 de Freitas, N. (2018). [Intrinsic Social Motivation via Causal Influence in Multi-Agent RL](https://arxiv.org/abs/1810.08647). arXiv preprint arXiv:1810.08647.\n        \n# Contributors\n\nThis code base was initially developed by Eugene Vinitsky and Natasha Jaques; help with reproduction was provided by Joel Leibo, Antonio Castenada, and Edward Hughes. Additional development was done by Hugo Heemskerk. Support for PettingZoo was provided by Rohan Potdar.\n\n# Citation\n\nIf you want to cite this repository accademic work, please use the following citation:\n\n```\n@misc{SSDOpenSource,\nauthor = {[Vinitsky, Eugene and Jaques, Natasha and Leibo, Joel and Castenada, Antonio and Hughes, Edward]},\ntitle = {An Open Source Implementation of Sequential Social Dilemma Games},\nyear = {2019},\npublisher = {GitHub},\nnote = {GitHub repository},\nhowpublished = {\\url{https://github.com/eugenevinitsky/sequential_social_dilemma_games/issues/182}}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftimf34%2Fssd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftimf34%2Fssd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftimf34%2Fssd/lists"}