{"id":26700493,"url":"https://github.com/google-deepmind/ai-safety-gridworlds","last_synced_at":"2025-03-27T00:28:21.324Z","repository":{"id":27074232,"uuid":"109390542","full_name":"google-deepmind/ai-safety-gridworlds","owner":"google-deepmind","description":"This is a suite of reinforcement learning environments illustrating various safety properties of intelligent agents.","archived":true,"fork":false,"pushed_at":"2022-05-18T19:19:21.000Z","size":110,"stargazers_count":593,"open_issues_count":0,"forks_count":117,"subscribers_count":49,"default_branch":"master","last_synced_at":"2024-04-16T04:53:36.768Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google-deepmind.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-11-03T12:00:12.000Z","updated_at":"2024-04-11T22:02:13.000Z","dependencies_parsed_at":"2022-07-27T09:22:06.297Z","dependency_job_id":null,"html_url":"https://github.com/google-deepmind/ai-safety-gridworlds","commit_stats":null,"previous_names":["google-deepmind/ai-safety-gridworlds"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-deepmind%2Fai-safety-gridworlds","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-deepmind%2Fai-safety-gridworlds/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-deepmind%2Fai-safety-gridworlds/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-deepmind%2Fai-safety-gridworlds/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google-deepmind","download_url":"https://codeload.github.com/google-deepmind/ai-safety-gridworlds/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245759283,"owners_count":20667691,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-27T00:28:19.693Z","updated_at":"2025-03-27T00:28:21.317Z","avatar_url":"https://github.com/google-deepmind.png","language":"Python","funding_links":[],"categories":["AI Incident Sources"],"sub_categories":["MITRE ATLAS and OWASP Relationships Diagram"],"readme":"# AI safety gridworlds\n\nThis is a suite of reinforcement learning environments illustrating various\nsafety properties of intelligent agents. These environments are\nimplemented in [pycolab](https://github.com/deepmind/pycolab), a\nhighly-customisable gridworld game engine with some batteries included.\n\nFor more information, see the accompanying [research\npaper](https://arxiv.org/pdf/1711.09883.pdf).\n\nFor the latest list of changes, see [CHANGES.md](https://github.com/deepmind/ai-safety-gridworlds/blob/master/CHANGES.md).\n\n## Instructions\n\n1.  Open a new terminal window (`iterm2` on Mac, `gnome-terminal` or `xterm` on\n    linux work best, avoid `tmux`/`screen`).\n2.  Set the terminal colours to `xterm-256color` by running `export\n    TERM=xterm-256color`.\n3.  Clone the repository using\n    `git clone https://github.com/deepmind/ai-safety-gridworlds.git`.\n4.  Choose an environment from the list below and run it by typing\n    `PYTHONPATH=. python -B ai_safety_gridworlds/environments/ENVIRONMENT_NAME.py`.\n\n## Dependencies\n\n* Python 2 (with enum34 support) or Python 3. We tested it with all the commonly used Python minor versions (2.7, 3.4, 3.5, 3.6). Note that the version 2.7.15 might have curses rendering issues in a terminal.\n* [Pycolab](https://github.com/deepmind/pycolab) which is the gridworlds game engine we use.\n* Numpy. Our version is 1.14.5. Note that the higher versions don't work with pip tensorflow at the moment.\n* [Abseil](https://github.com/abseil/abseil-py) Python common libraries.\n* If you intend to contribute and run the test suite, you will also need Tensorflow, as pycolab relies on it for testing.\n\nWe also recommend using a virtual environment. Under the assumption that you have the virtualenv package installed, the setup is as follows.\n\nFor python2:\n```\nvirtualenv py2\n. ./py2/bin/activate\npip install absl-py numpy pycolab enum34 tensorflow\n```\n\nFor python3:\n```\nvirtualenv -p /usr/bin/python3 py3\n. ./py3/bin/activate\npip install absl-py numpy pycolab tensorflow\n```\n\n\n## Environments\n\nOur suite includes the following environments.\n\n1.  **Safe interruptibility**: We want to be able to interrupt an agent and\n    override its actions at any time. How can we prevent the agent from learning\n    to avoid interruptions? `safe_interruptibility.py`\n2.  **Avoiding side effects**: How can we incentivize agents to minimize effects\n    unrelated to their main objectives, especially those that are irreversible\n    or difficult to reverse? `side_effects_sokoban.py` and `conveyor_belt.py`\n3.  **Absent supervisor**: How can we ensure that the agent does not behave\n    differently depending on whether it is being supervised?\n    `absent_supervisor.py`\n4.  **Reward gaming**: How can we design agents that are robust to misspecified\n    reward functions, for example by modeling their uncertainty about the reward\n    function? `boat_race.py` and `tomato_watering.py`\n5.  **Self-modification**: Can agents be robust to limited self-modifications,\n    for example if they can increase their exploration rate? `whisky-gold.py`\n6.  **Distributional shift**: How can we detect and adapt to a data distribution\n    that is different from the training distribution? `distributional_shift.py`\n7.  **Robustness to adversaries**: How can we ensure the agent's performance\n    does not degrade in the presence of adversaries? `friend_foe.py`\n8.  **Safe exploration**: How can we ensure satisfying a safety constraint under\n    unknown environment dynamics? `island_navigation.py`\n\nOur environments are Markov Decision Processes. All environments use a grid of\nsize at most 10x10. Each cell in the grid can be empty, or contain a wall or\nother objects. These objects are specific to each environment and are explained\nin the corresponding section in the paper. The agent is located in one cell on\nthe grid and in every step the agent takes one of the actions from the action\nset A = {left, right, up, down}. Each action modifies the agent's position to\nthe next cell in the corresponding direction unless that cell is a wall or\nanother impassable object, in which case the agent stays put.\n\nThe agent interacts with the environment in an episodic setting: at the start of\neach episode, the environment is reset to its starting configuration (which is\npossibly randomized). The agent then interacts with the environment until the\nepisode ends, which is specific to each environment. We fix the maximal episode\nlength to 100 steps. Several environments contain a goal cell, depicted as G. If\nthe agent enters the goal cell, it receives a reward of +50 and the episode\nends. We also provide a default reward of −1 in every time-step to encourage\nfinishing the episode sooner than later, and use no discounting in the\nenvironment.\n\nIn the classical reinforcement learning framework, the agent's objective is to\nmaximize the cumulative (visible) reward signal. While this is an important part\nof the agent's objective, in some problems this does not capture everything that\nwe care about. Instead of the reward function, we evaluate the agent on the\nperformance function *that is not observed by the agent*. The performance\nfunction might or might not be identical to the reward function. In real-world\nexamples, the performance function would only be implicitly defined by the\ndesired behavior the human designer wishes to achieve, but is inaccessible to\nthe agent and the human designer.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-deepmind%2Fai-safety-gridworlds","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle-deepmind%2Fai-safety-gridworlds","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-deepmind%2Fai-safety-gridworlds/lists"}