{"id":15654956,"url":"https://github.com/jvmncs/safe-grid-agents","last_synced_at":"2025-08-19T09:13:29.212Z","repository":{"id":80109998,"uuid":"130503732","full_name":"jvmncs/safe-grid-agents","owner":"jvmncs","description":"Training (hopefully) safe agents in gridworlds","archived":false,"fork":false,"pushed_at":"2019-05-12T20:14:20.000Z","size":227,"stargazers_count":25,"open_issues_count":7,"forks_count":4,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-05-01T14:37:00.355Z","etag":null,"topics":["gridworld","gym","reinforcement-learning","safe-agents","safe-reinforcement-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jvmncs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-04-21T19:00:18.000Z","updated_at":"2021-10-26T15:24:52.000Z","dependencies_parsed_at":"2024-01-20T23:15:08.426Z","dependency_job_id":null,"html_url":"https://github.com/jvmncs/safe-grid-agents","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jvmncs/safe-grid-agents","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jvmncs%2Fsafe-grid-agents","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jvmncs%2Fsafe-grid-agents/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jvmncs%2Fsafe-grid-agents/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jvmncs%2Fsafe-grid-agents/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jvmncs","download_url":"https://codeload.github.com/jvmncs/safe-grid-agents/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jvmncs%2Fsafe-grid-agents/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271128945,"owners_count":24703879,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-19T02:00:09.176Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gridworld","gym","reinforcement-learning","safe-agents","safe-reinforcement-learning"],"created_at":"2024-10-03T12:55:14.115Z","updated_at":"2025-08-19T09:13:29.192Z","avatar_url":"https://github.com/jvmncs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# safe-grid-agents\n\nTraining (hopefully) safe agents in gridworlds.\n\nEmphasizing extensibility, modularity, and accessibility.\n\n## Layout\n\n-   `safe_grid_agents/common`: Core codebase. Includes abstract base\n    classes for a variety of agents, their associated warmup/learn/eval\n    functions, and a utilities file.\n-   `main.py`: Python executable for composing training jobs.\n-   `safe_grid_agents/parsing`: Helpers that construct a flexible CLI\n    for `main.py`.\n-   `safe_grid_agents/ssrl`: Agents that implement semi-supervised\n    reinforcement learning and their associated warmup functions.\n\n## Installation\n\nWhen installing with pip, make sure to use the\n`process-dependency-links` flag:\n\n``` {.sh}\npip install . --process-dependency-links\n```\n\nURL-based dependencies are available for audit at the following\nrepositories and forks: -\n[safe-grid-gym](https://github.com/jvmancuso/safe-grid-gym) -\n[ai-safety-gridworlds](https://github.com/jvmancuso/ai-safety-gridworlds)\n\nIf you plan on developing this library, make sure to add an `-e` flag to\nthe above pip install command.\n\nThis repo requires [tensorboardX](https://github.com/lanpa/tensorboardX)\nfor monitoring and visualizing agent learning, as well as PyTorch for\nimplementation of certain agents. Currently, tensorboardX does not\nfunction properly without Tensorflow installed. Since the installation\nprocess of these packages can vary system to system, we exclude them\nfrom our build process. There are multiple tutorials online for\ninstalling both of these online. For example, on OS X without CUDA\nsupport I'd go with:\n\n``` {.sh}\n# Replace `tensorflow` with `tensorflow-gpu` if you have a GPU.\npip install torch torchvision tensorflow\n```\n\n# Usage\n\n## Training agents\n\nYou can use the CLI to `main.py` to modularly drop agents into arbitrary\nsafety gridworlds. For example, `python main.py boat tabular-q --lr .5`\nwill train a TabularQAgent on the BoatRaceEnvironment with a learning\nrate of 0.5.\n\nThere are a number of customizable parameters to modify training runs.\nThese parameters are split into three groups: - Core arguments: args\nthat are shared across all agents/environments. Found in\n[`parsing/core_parser_configs.yaml`](https://github.com/jvmancuso/safe-grid-agents/blob/master/safe_grid_agents/parsing/core_parser_configs.yaml).\n- Environment arguments: args specific to environments but shared across\nagents. Currently empty, but could be useful for specific environments,\ndepending on the agent. Found in\n[`parsing/env_parser_configs.yaml`](https://github.com/jvmancuso/safe-grid-agents/blob/master/safe_grid_agents/parsing/env_parser_configs.yaml).\n- Agent environments: args specific to agents. Most hyperparameters live\nhere. Found in\n[`parsing/agent_parser_configs.yaml`](https://github.com/jvmancuso/safe-grid-agents/blob/master/safe_grid_agents/parsing/agent_parser_configs.yaml).\n\nThe generalized form for the CLI is\n\n``` {.sh}\npython main.py \u003ccore_args\u003e env \u003cenv_args\u003e agent \u003cagent_args\u003e\n```\n\n## Ray Tune\n\nWe support using Ray Tune to configure hyperparameters. Look at\n`TUNE_DEFAULT_CONFIG` in `main.py` to see which are currently supported.\nIf you specify a tunable parameter on the CLI with the `-t` or `--tune`\nflag, it will be automatically set.\n\n### Example\n\nThis will automatically set parameters for the learning rate `lr` and\ndiscount rate `discount`.\n\n``` {.sh}\n# `-t` and `--tune` are equivalent, and can be used interchangeably.\npython3 main.py -t lr --tune discount boat tabular-q\n```\n\n## Monitoring agent learning with tensorboardX\n\nYou can use the `--log-dir`/`-L` flag to the main.py script to specify a\ndirectory for saving training and evaluation metrics across runs. I\nsuggest a pattern similar to\n\n``` {.sh}\nlogs/sokoban/deep-q/lr5e-4\n# that is, \u003clogdir\u003e/\u003cenv_alias\u003e/\u003cagent_alias\u003e/\u003cuniqueid_or_hparams\u003e\n```\n\nIf no log-dir is specified for main.py, logging defaults to the `runs/`\ndirectory, which can be helpful to separate debugging runs from training\nruns.\n\nGiven a log directory `\u003clogs\u003e`, simply run `tensorboard --logdir \u003clogs\u003e`\nto visualize an agent's learning.\n\n# Development\n\n## Code style\n\nWe use [black](https://github.com/ambv/black) for auto-formatting\naccording to a consistent style guide. To auto format, run `black .`\nfrom inside the repo folder. To make this more convenient, you can\ninstall plugins for your preferred text editor that auto-format on every\nsave.\n\n## Adding agents\n\nSteps to take when adding a new agent.\n\n1.  Determine where the agent should live; for example, if you're\n    testing a new baseline from standard RL, include it in `common`, but\n    if you're adding a new SSRL agent, add it to `ssrl`. We'll refer to\n    this folder as `\u003ctop\u003e`.\n2.  (optional) If your agent doesn't fall into these categories, create\n    a new top-level subdirectory `\u003ctop\u003e` for it (using an informative\n    abbreviation). You should also create an abstract base class\n    establishing the distinguishing functionality of your agent class in\n    `\u003ctop\u003e/base.py`. For example:\n    -   SSRL requires a stronger agent H to learn from, so we require a\n        `query_H` method for each agent.\n    -   Additionally, following [Everitt et\n        al.](https://arxiv.org/abs/1705.08417), we require a `learn_C`\n        method to learn the probability of the state being corrupt.\n3.  (optional) Implement a warmup function in `\u003ctop\u003e/warmup.py`, and\n    make sure it's importable from `common/warmup.py`. The `noop`\n    default warmup function works for agents that don't require any\n    special functionality.\n4.  Implement a function describing the agent's learning feedback loop\n    in `\u003ctop\u003e/learn.py`. See\n    [`common/learn.py`](https://github.com/jvmancuso/safe-grid-agents/blob/master/safe_grid_agents/common/learn.py)\n    for an example distinguishing DQN from a tabular Q-learning agent.\n5.  (optional) Implement a function in `\u003ctop\u003e/eval.py` describing the\n    evaluation feedback loop. The `default_eval` function in\n    `common/eval.py` should cover most cases, so you may not need to add\n    anything for evaluation.\n6.  Add a new entry for the agent's CLI arguments in\n    `parsing/agent_parser_configs.yaml`. Follow the existing pattern and\n    check for previously implemented YAML anchors that cover the\n    arguments you need (e.g. `learnrate`, `epsilon-anneal`, etc.). These\n    configs should be organized by where they appear in the folder\n    structure of the repository.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjvmncs%2Fsafe-grid-agents","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjvmncs%2Fsafe-grid-agents","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjvmncs%2Fsafe-grid-agents/lists"}