{"id":15176724,"url":"https://github.com/intellabs/coach","last_synced_at":"2025-10-01T16:30:40.378Z","repository":{"id":45258679,"uuid":"105468219","full_name":"IntelLabs/coach","owner":"IntelLabs","description":"Reinforcement Learning Coach by Intel AI Lab enables easy experimentation with state of the art Reinforcement Learning algorithms","archived":true,"fork":false,"pushed_at":"2022-12-11T17:54:07.000Z","size":75315,"stargazers_count":2306,"open_issues_count":90,"forks_count":461,"subscribers_count":127,"default_branch":"master","last_synced_at":"2024-04-23T13:03:54.807Z","etag":null,"topics":["carla","coach","deep-learning","distributed-reinforcement-learning","hierarchical-reinforcement-learning","imitation-learning","mujoco","mxnet","onnx","openai-gym","reinforcement-learning","rl","roboschool","starcraft","starcraft2","starcraft2-ai","tensorflow"],"latest_commit_sha":null,"homepage":"https://intellabs.github.io/coach/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/IntelLabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-10-01T19:27:43.000Z","updated_at":"2024-04-16T02:52:18.000Z","dependencies_parsed_at":"2023-01-27T04:16:17.896Z","dependency_job_id":null,"html_url":"https://github.com/IntelLabs/coach","commit_stats":null,"previous_names":["nervanasystems/coach"],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IntelLabs%2Fcoach","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IntelLabs%2Fcoach/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IntelLabs%2Fcoach/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IntelLabs%2Fcoach/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/IntelLabs","download_url":"https://codeload.github.com/IntelLabs/coach/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234882553,"owners_count":18901291,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["carla","coach","deep-learning","distributed-reinforcement-learning","hierarchical-reinforcement-learning","imitation-learning","mujoco","mxnet","onnx","openai-gym","reinforcement-learning","rl","roboschool","starcraft","starcraft2","starcraft2-ai","tensorflow"],"created_at":"2024-09-27T13:41:01.906Z","updated_at":"2025-10-01T16:30:37.757Z","avatar_url":"https://github.com/IntelLabs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003e :warning: **DISCONTINUATION OF PROJECT** - \n\u003e *This project will no longer be maintained by Intel.\n\u003e Intel has ceased development and contributions including, but not limited to, maintenance, bug fixes, new releases, or updates, to this project.*\n\u003e **Intel no longer accepts patches to this project.**\n\u003e *If you have an ongoing need to use this project, are interested in independently developing it, or would like to maintain patches for the open source software community, please create your own fork of this project.*\n\n# Coach\n\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/IntelLabs/coach/blob/master/LICENSE)\n[![Docs](https://readthedocs.org/projects/carla/badge/?version=latest)](https://intellabs.github.io/coach/)\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.1134898.svg)](https://doi.org/10.5281/zenodo.1134898)\n[![Downloads](https://static.pepy.tech/personalized-badge/rl-coach?period=total\u0026units=international_system\u0026left_color=grey\u0026right_color=blue\u0026left_text=rl-coach%20downloads)](https://pepy.tech/project/rl-coach)\n[![Downloads](https://static.pepy.tech/personalized-badge/rl-coach-slim?period=total\u0026units=international_system\u0026left_color=grey\u0026right_color=blue\u0026left_text=rl-coach-slim%20downloads)](https://pepy.tech/project/rl-coach-slim)\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"img/coach_logo.png\" alt=\"Coach Logo\" width=\"200\"/\u003e\u003c/p\u003e\n\nCoach is a python reinforcement learning framework containing implementation of many state-of-the-art algorithms.\n\nIt exposes a set of easy-to-use APIs for experimenting with new RL algorithms, and allows simple integration of new environments to solve. \nBasic RL components (algorithms, environments, neural network architectures, exploration policies, ...) are well decoupled, so that extending and reusing existing components is fairly painless.\n\nTraining an agent to solve an environment is as easy as running:\n\n```bash\ncoach -p CartPole_DQN -r\n```\n\n\u003cimg src=\"img/fetch_slide.gif\" alt=\"Fetch Slide\"/\u003e \u003cimg src=\"img/pendulum.gif\" alt=\"Pendulum\"/\u003e \u003cimg src=\"img/starcraft.gif\" width = \"281\" height =\"200\" alt=\"Starcraft\"/\u003e\n\u003cbr\u003e\n\u003cimg src=\"img/doom_deathmatch.gif\" alt=\"Doom Deathmatch\"/\u003e \u003cimg src=\"img/carla.gif\" alt=\"CARLA\"/\u003e \u003cimg src=\"img/montezuma.gif\" alt=\"MontezumaRevenge\" width = \"164\" height =\"200\"/\u003e\n\u003cbr\u003e\n\u003cimg src=\"img/doom_health.gif\" alt=\"Doom Health Gathering\"/\u003e \u003cimg src=\"img/minitaur.gif\" alt=\"PyBullet Minitaur\" width = \"249\" height =\"200\"/\u003e \u003cimg src=\"img/ant.gif\" alt=\"Gym Extensions Ant\"/\u003e\n\u003cbr\u003e\u003cbr\u003e\n\n* [Release 0.8.0](https://ai.intel.com/reinforcement-learning-coach-intel/) (initial release)\n* [Release 0.9.0](https://ai.intel.com/reinforcement-learning-coach-carla-qr-dqn/)\n* [Release 0.10.0](https://ai.intel.com/introducing-reinforcement-learning-coach-0-10-0/)\n* [Release 0.11.0](https://ai.intel.com/rl-coach-data-science-at-scale)\n* [Release 0.12.0](https://github.com/IntelLabs/coach/releases/tag/v0.12.0) \n* [Release 1.0.0](https://www.intel.ai/rl-coach-new-release) (current release)\n\n\n## Table of Contents\n\n- [Benchmarks](#benchmarks)\n- [Installation](#installation)\n- [Getting Started](#getting-started)\n  * [Tutorials and Documentation](#tutorials-and-documentation)\n  * [Basic Usage](#basic-usage)\n    * [Running Coach](#running-coach)\n    * [Running Coach Dashboard (Visualization)](#running-coach-dashboard-visualization)\n  * [Distributed Multi-Node Coach](#distributed-multi-node-coach)\n  * [Batch Reinforcement Learning](#batch-reinforcement-learning)\n- [Supported Environments](#supported-environments)\n  * [Note on MuJoCo version](#note-on-mujoco-version)\n- [Supported Algorithms](#supported-algorithms)\n- [Citation](#citation)\n- [Contact](#contact)\n- [Disclaimer](#disclaimer)\n\n## Benchmarks\n\nOne of the main challenges when building a research project, or a solution based on a published algorithm, is getting a concrete and reliable baseline that reproduces the algorithm's results, as reported by its authors. To address this problem, we are releasing a set of [benchmarks](benchmarks) that shows Coach reliably reproduces many state of the art algorithm results.\n\n## Installation\n\nNote: Coach has only been tested on Ubuntu 16.04 LTS, and with Python 3.5.\n\nFor some information on installing on Ubuntu 17.10 with Python 3.6.3, please refer to the following issue: https://github.com/IntelLabs/coach/issues/54\n\nIn order to install coach, there are a few prerequisites required. This will setup all the basics needed to get the user going with running Coach on top of [OpenAI Gym](https://github.com/openai/gym) environments:\n\n```\n# General\nsudo -E apt-get install python3-pip cmake zlib1g-dev python3-tk python-opencv -y\n\n# Boost libraries\nsudo -E apt-get install libboost-all-dev -y\n\n# Scipy requirements\nsudo -E apt-get install libblas-dev liblapack-dev libatlas-base-dev gfortran -y\n\n# PyGame\nsudo -E apt-get install libsdl-dev libsdl-image1.2-dev libsdl-mixer1.2-dev libsdl-ttf2.0-dev\nlibsmpeg-dev libportmidi-dev libavformat-dev libswscale-dev -y\n\n# Dashboard\nsudo -E apt-get install dpkg-dev build-essential python3.5-dev libjpeg-dev  libtiff-dev libsdl1.2-dev libnotify-dev \nfreeglut3 freeglut3-dev libsm-dev libgtk2.0-dev libgtk-3-dev libwebkitgtk-dev libgtk-3-dev libwebkitgtk-3.0-dev\nlibgstreamer-plugins-base1.0-dev -y\n\n# Gym\nsudo -E apt-get install libav-tools libsdl2-dev swig cmake -y\n```\n\nWe recommend installing coach in a virtualenv:\n\n```\nsudo -E pip3 install virtualenv\nvirtualenv -p python3 coach_env\n. coach_env/bin/activate\n```\n\nFinally, install coach using pip:\n```\npip3 install rl_coach\n```\n\nOr alternatively, for a development environment, install coach from the cloned repository:\n```\ncd coach\npip3 install -e .\n```\n\nIf a GPU is present, Coach's pip package will install tensorflow-gpu, by default. If a GPU is not present, an [Intel-Optimized TensorFlow](https://software.intel.com/en-us/articles/intel-optimized-tensorflow-wheel-now-available), will be installed. \n\nIn addition to OpenAI Gym, several other environments were tested and are supported. Please follow the instructions in the Supported Environments section below in order to install more environments.\n\n## Getting Started\n\n### Tutorials and Documentation\n[Jupyter notebooks demonstrating how to run Coach from command line or as a library, implement an algorithm, or integrate an environment](https://github.com/IntelLabs/coach/tree/master/tutorials).\n\n[Framework documentation, algorithm description and instructions on how to contribute a new agent/environment](https://intellabs.github.io/coach/).\n\n### Basic Usage\n\n#### Running Coach\n\nTo allow reproducing results in Coach, we defined a mechanism called _preset_. \nThere are several available presets under the `presets` directory.\nTo list all the available presets use the `-l` flag.\n\nTo run a preset, use:\n\n```bash\ncoach -r -p \u003cpreset_name\u003e\n```\n\nFor example:\n* CartPole environment using Policy Gradients (PG):\n\n  ```bash\n  coach -r -p CartPole_PG\n  ```\n  \n* Basic level of Doom using Dueling network and Double DQN (DDQN) algorithm:\n\n  ```bash\n  coach -r -p Doom_Basic_Dueling_DDQN\n  ```\n\nSome presets apply to a group of environment levels, like the entire Atari or Mujoco suites for example.\nTo use these presets, the requeseted level should be defined using the `-lvl` flag.\n\nFor example:\n\n\n* Pong using the Neural Episodic Control (NEC) algorithm:\n\n  ```bash\n  coach -r -p Atari_NEC -lvl pong\n  ```\n\nThere are several types of agents that can benefit from running them in a distributed fashion with multiple workers in parallel. Each worker interacts with its own copy of the environment but updates a shared network, which improves the data collection speed and the stability of the learning process.\nTo specify the number of workers to run, use the `-n` flag.\n\nFor example:\n* Breakout using Asynchronous Advantage Actor-Critic (A3C) with 8 workers:\n\n  ```bash\n  coach -r -p Atari_A3C -lvl breakout -n 8\n  ```\n\n\nIt is easy to create new presets for different levels or environments by following the same pattern as in presets.py\n\nMore usage examples can be found [here](https://github.com/IntelLabs/coach/blob/master/tutorials/0.%20Quick%20Start%20Guide.ipynb).\n\n#### Running Coach Dashboard (Visualization)\nTraining an agent to solve an environment can be tricky, at times. \n\nIn order to debug the training process, Coach outputs several signals, per trained algorithm, in order to track algorithmic performance. \n\nWhile Coach trains an agent, a csv file containing the relevant training signals will be saved to the 'experiments' directory. Coach's dashboard can then be used to dynamically visualize the training signals, and track algorithmic behavior. \n\nTo use it, run:\n\n```bash\ndashboard\n```\n\n\n\n\u003cimg src=\"img/dashboard.gif\" alt=\"Coach Design\" style=\"width: 800px;\"/\u003e\n\n\n### Distributed Multi-Node Coach\n\nAs of release 0.11.0, Coach supports horizontal scaling for training RL agents on multiple nodes. In release 0.11.0 this was tested on the ClippedPPO and DQN agents.\nFor usage instructions please refer to the documentation [here](https://intellabs.github.io/coach/dist_usage.html).\n\n### Batch Reinforcement Learning\n\nTraining and evaluating an agent from a dataset of experience, where no simulator is available, is supported in Coach. \nThere are [example](https://github.com/IntelLabs/coach/blob/master/rl_coach/presets/CartPole_DDQN_BatchRL.py) [presets](https://github.com/IntelLabs/coach/blob/master/rl_coach/presets/Acrobot_DDQN_BCQ_BatchRL.py) and a [tutorial](https://github.com/IntelLabs/coach/blob/master/tutorials/4.%20Batch%20Reinforcement%20Learning.ipynb). \n\n\n## Supported Environments\n\n* *OpenAI Gym:*\n\n    Installed by default by Coach's installer (see note on MuJoCo version [below](#note-on-mujoco-version)).\n\n* *ViZDoom:*\n\n    Follow the instructions described in the ViZDoom repository -\n\n    https://github.com/mwydmuch/ViZDoom\n\n    Additionally, Coach assumes that the environment variable VIZDOOM_ROOT points to the ViZDoom installation directory.\n\n* *Roboschool:*\n\n    Follow the instructions described in the roboschool repository - \n\n    https://github.com/openai/roboschool\n\n* *GymExtensions:*\n\n    Follow the instructions described in the GymExtensions repository -\n\n    https://github.com/Breakend/gym-extensions\n\n    Additionally, add the installation directory to the PYTHONPATH environment variable.\n\n* *PyBullet:*\n\n    Follow the instructions described in the [Quick Start Guide](https://docs.google.com/document/d/10sXEhzFRSnvFcl3XxNGhnD4N2SedqwdAvK3dsihxVUA) (basically just - 'pip install pybullet')\n\n* *CARLA:*\n\n    Download release 0.8.4 from the CARLA repository -\n\n    https://github.com/carla-simulator/carla/releases\n\n    Install the python client and dependencies from the release tarball:\n    ```\n    pip3 install -r PythonClient/requirements.txt\n    pip3 install PythonClient\n    ```\n\n    Create a new CARLA_ROOT environment variable pointing to CARLA's installation directory.\n\n    A simple CARLA settings file (```CarlaSettings.ini```) is supplied with Coach, and is located in the ```environments``` directory.\n\n* *Starcraft:*\n\n    Follow the instructions described in the PySC2 repository - \n    \n    https://github.com/deepmind/pysc2\n    \n* *DeepMind Control Suite:*\n\n    Follow the instructions described in the DeepMind Control Suite repository - \n    \n    https://github.com/deepmind/dm_control\n\n* *Robosuite:*\u003ca name=\"robosuite\"\u003e\u003c/a\u003e\n\n    **__Note:__ To use Robosuite-based environments, please install Coach from the latest cloned repository. It is not yet available as part of the `rl_coach` package on PyPI.**\n    \n    Follow the instructions described in the [robosuite documentation](https://robosuite.ai/docs/installation.html) (see note on MuJoCo version [below](#note-on-mujoco-version)).\n\n### Note on MuJoCo version\n\nOpenAI Gym supports MuJoCo only up to version 1.5 (and corresponding mujoco-py version 1.50.x.x). The Robosuite simulation framework, however, requires MuJoCo version 2.0 (and corresponding mujoco-py version 2.0.2.9, as of robosuite version 1.2). Therefore, if you wish to run both Gym-based MuJoCo environments and Robosuite environments, it's recommended to have a separate virtual environment for each.\n\nPlease note that all Gym-Based MuJoCo presets in Coach (`rl_coach/presets/Mujoco_*.py`) have been validated _**only**_ with MuJoCo 1.5 (including the reported [benchmark results](benchmarks)).\n    \n\n## Supported Algorithms\n\n\u003cimg src=\"docs_raw/source/_static/img/algorithms.png\" alt=\"Coach Design\" style=\"width: 800px;\"/\u003e\n\n\n\n\n### Value Optimization Agents\n* [Deep Q Network (DQN)](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf)  ([code](rl_coach/agents/dqn_agent.py))\n* [Double Deep Q Network (DDQN)](https://arxiv.org/pdf/1509.06461.pdf)  ([code](rl_coach/agents/ddqn_agent.py))\n* [Dueling Q Network](https://arxiv.org/abs/1511.06581)\n* [Mixed Monte Carlo (MMC)](https://arxiv.org/abs/1703.01310)  ([code](rl_coach/agents/mmc_agent.py))\n* [Persistent Advantage Learning (PAL)](https://arxiv.org/abs/1512.04860)  ([code](rl_coach/agents/pal_agent.py))\n* [Categorical Deep Q Network (C51)](https://arxiv.org/abs/1707.06887)  ([code](rl_coach/agents/categorical_dqn_agent.py))\n* [Quantile Regression Deep Q Network (QR-DQN)](https://arxiv.org/pdf/1710.10044v1.pdf)  ([code](rl_coach/agents/qr_dqn_agent.py))\n* [N-Step Q Learning](https://arxiv.org/abs/1602.01783) | **Multi Worker Single Node**  ([code](rl_coach/agents/n_step_q_agent.py))\n* [Neural Episodic Control (NEC)](https://arxiv.org/abs/1703.01988)  ([code](rl_coach/agents/nec_agent.py))\n* [Normalized Advantage Functions (NAF)](https://arxiv.org/abs/1603.00748.pdf) | **Multi Worker Single Node**  ([code](rl_coach/agents/naf_agent.py))\n* [Rainbow](https://arxiv.org/abs/1710.02298)  ([code](rl_coach/agents/rainbow_dqn_agent.py))\n\n### Policy Optimization Agents\n* [Policy Gradients (PG)](http://www-anw.cs.umass.edu/~barto/courses/cs687/williams92simple.pdf) | **Multi Worker Single Node**  ([code](rl_coach/agents/policy_gradients_agent.py))\n* [Asynchronous Advantage Actor-Critic (A3C)](https://arxiv.org/abs/1602.01783) | **Multi Worker Single Node**  ([code](rl_coach/agents/actor_critic_agent.py))\n* [Deep Deterministic Policy Gradients (DDPG)](https://arxiv.org/abs/1509.02971) | **Multi Worker Single Node**  ([code](rl_coach/agents/ddpg_agent.py))\n* [Proximal Policy Optimization (PPO)](https://arxiv.org/pdf/1707.06347.pdf)  ([code](rl_coach/agents/ppo_agent.py))\n* [Clipped Proximal Policy Optimization (CPPO)](https://arxiv.org/pdf/1707.06347.pdf) | **Multi Worker Single Node**  ([code](rl_coach/agents/clipped_ppo_agent.py))\n* [Generalized Advantage Estimation (GAE)](https://arxiv.org/abs/1506.02438) ([code](rl_coach/agents/actor_critic_agent.py#L86))\n* [Sample Efficient Actor-Critic with Experience Replay (ACER)](https://arxiv.org/abs/1611.01224) | **Multi Worker Single Node**  ([code](rl_coach/agents/acer_agent.py))\n* [Soft Actor-Critic (SAC)](https://arxiv.org/abs/1801.01290) ([code](rl_coach/agents/soft_actor_critic_agent.py))\n* [Twin Delayed Deep Deterministic Policy Gradient (TD3)](https://arxiv.org/pdf/1802.09477.pdf) ([code](rl_coach/agents/td3_agent.py))\n\n### General Agents\n* [Direct Future Prediction (DFP)](https://arxiv.org/abs/1611.01779) | **Multi Worker Single Node**  ([code](rl_coach/agents/dfp_agent.py))\n\n### Imitation Learning Agents\n* Behavioral Cloning (BC)  ([code](rl_coach/agents/bc_agent.py))\n* [Conditional Imitation Learning](https://arxiv.org/abs/1710.02410) ([code](rl_coach/agents/cil_agent.py))\n\n### Hierarchical Reinforcement Learning Agents\n* [Hierarchical Actor Critic (HAC)](https://arxiv.org/abs/1712.00948.pdf) ([code](rl_coach/agents/hac_ddpg_agent.py))\n\n### Memory Types\n* [Hindsight Experience Replay (HER)](https://arxiv.org/abs/1707.01495.pdf) ([code](rl_coach/memories/episodic/episodic_hindsight_experience_replay.py))\n* [Prioritized Experience Replay (PER)](https://arxiv.org/abs/1511.05952) ([code](rl_coach/memories/non_episodic/prioritized_experience_replay.py))\n\n### Exploration Techniques\n* E-Greedy ([code](rl_coach/exploration_policies/e_greedy.py))\n* Boltzmann ([code](rl_coach/exploration_policies/boltzmann.py))\n* Ornstein–Uhlenbeck process ([code](rl_coach/exploration_policies/ou_process.py))\n* Normal Noise ([code](rl_coach/exploration_policies/additive_noise.py))\n* Truncated Normal Noise ([code](rl_coach/exploration_policies/truncated_normal.py))\n* [Bootstrapped Deep Q Network](https://arxiv.org/abs/1602.04621)  ([code](rl_coach/agents/bootstrapped_dqn_agent.py))\n* [UCB Exploration via Q-Ensembles (UCB)](https://arxiv.org/abs/1706.01502) ([code](rl_coach/exploration_policies/ucb.py))\n* [Noisy Networks for Exploration](https://arxiv.org/abs/1706.10295) ([code](rl_coach/exploration_policies/parameter_noise.py))\n\n## Citation\n\nIf you used Coach for your work, please use the following citation:\n\n```\n@misc{caspi_itai_2017_1134899,\n  author       = {Caspi, Itai and\n                  Leibovich, Gal and\n                  Novik, Gal and\n                  Endrawis, Shadi},\n  title        = {Reinforcement Learning Coach},\n  month        = dec,\n  year         = 2017,\n  doi          = {10.5281/zenodo.1134899},\n  url          = {https://doi.org/10.5281/zenodo.1134899}\n}\n```\n\n## Contact\n\nWe'd be happy to get any questions or contributions through GitHub issues and PRs.\n\nPlease make sure to take a look [here](CONTRIBUTING.md) before filing an issue or proposing a PR.\n\nThe Coach development team can also be contacted over [email](mailto:coach@intel.com)\n\n\n## Disclaimer\n\nCoach is released as a reference code for research purposes. It is not an official Intel product, and the level of quality and support may not be as expected from an official product. \nAdditional algorithms and environments are planned to be added to the framework. Feedback and contributions from the open source and RL research communities are more than welcome.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fintellabs%2Fcoach","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fintellabs%2Fcoach","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fintellabs%2Fcoach/lists"}