{"id":28753253,"url":"https://github.com/google-deepmind/dqn_zoo","last_synced_at":"2025-06-17T00:39:20.967Z","repository":{"id":41037286,"uuid":"297634132","full_name":"google-deepmind/dqn_zoo","owner":"google-deepmind","description":"DQN Zoo is a collection of reference implementations of reinforcement learning agents developed at DeepMind based on the Deep Q-Network (DQN) agent.","archived":false,"fork":false,"pushed_at":"2024-04-06T11:24:09.000Z","size":25874,"stargazers_count":472,"open_issues_count":2,"forks_count":82,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-05-18T21:02:47.479Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google-deepmind.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-09-22T11:57:54.000Z","updated_at":"2025-05-10T13:30:16.000Z","dependencies_parsed_at":"2025-05-18T21:12:53.881Z","dependency_job_id":null,"html_url":"https://github.com/google-deepmind/dqn_zoo","commit_stats":null,"previous_names":["google-deepmind/dqn_zoo","deepmind/dqn_zoo"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/google-deepmind/dqn_zoo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-deepmind%2Fdqn_zoo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-deepmind%2Fdqn_zoo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-deepmind%2Fdqn_zoo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-deepmind%2Fdqn_zoo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google-deepmind","download_url":"https://codeload.github.com/google-deepmind/dqn_zoo/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-deepmind%2Fdqn_zoo/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260268635,"owners_count":22983601,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-17T00:39:19.989Z","updated_at":"2025-06-17T00:39:20.907Z","avatar_url":"https://github.com/google-deepmind.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DQN Zoo\n\n*DQN Zoo* is a collection of reference implementations of reinforcement learning\nagents developed at DeepMind based on the\n[Deep Q-Network (DQN)](http://www.nature.com/articles/nature14236) agent.\n\nIt aims to be research-friendly, self-contained and readable. Each agent is\nimplemented using [JAX](http://github.com/google/jax),\n[Haiku](http://github.com/deepmind/haiku) and\n[RLax](http://github.com/deepmind/rlax), and is a best-effort replication of the\ncorresponding paper implementation. Each agent reproduces results on the\nstandard set of 57 Atari games, on average.\n\n\u003c!-- mdformat off(for readability) --\u003e\n\n| Directory     | Paper                                                                                                    |\n| ------------- | -------------------------------------------------------------------------------------------------------- |\n| `dqn`         | [Human Level Control Through Deep Reinforcement Learning](http\\://www.nature.com/articles/nature14236)   |\n| `double_q`    | [Deep Reinforcement Learning with Double Q-learning](http\\://arxiv.org/abs/1509.06461)                   |\n| `prioritized` | [Prioritized Experience Replay](http\\://arxiv.org/abs/1511.05952)                                        |\n| `c51`         | [A Distributional Perspective on Reinforcement Learning](http\\://arxiv.org/abs/1707.06887)               |\n| `qrdqn`       | [Distributional Reinforcement Learning with Quantile Regression](http\\://arxiv.org/abs/1710.10044)       |\n| `rainbow`     | [Rainbow: Combining Improvements in Deep Reinforcement Learning](http\\://arxiv.org/abs/1710.02298)       |\n| `iqn`         | [Implicit Quantile Networks for Distributional Reinforcement Learning](http\\://arxiv.org/abs/1806.06923) |\n\n\u003c!-- mdformat on --\u003e\n\nPlot of median human-normalized score over all 57 Atari games for each agent:\n\n![Plot summary](plot_atari_summary.svg)\n\n## Quick start\n\nNOTE: Only Python 3.9 and above and Linux is supported.\n\nFollow these steps to quickly clone the DQN Zoo repository, install all required\ndependencies and start running DQN. Prerequisites for these steps are a NVIDIA\nGPU with recent CUDA drivers.\n\n\u003c!-- mdlint off() --\u003e\n\n1.  Install [Docker](http://docs.docker.com/) version 19.03 or later (for the\n    `--gpus` flag).\n1.  Install [NVIDIA Container Toolkit](http://github.com/NVIDIA/nvidia-docker).\n1.  Enable\n    [sudoless docker](http://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user).\n\n1.  Verify the previous steps were successful e.g. by running: \\\n    `docker run --gpus all --rm nvidia/cuda:11.1.1-base nvidia-smi`\n\n1.  Download the script [`run.sh`](run.sh). This automatically downloads the\n    Atari ROMs from http://www.atarimania.com. The ROMs are available here for\n    free but make sure the respective license covers your particular use case.\n\nRunning this script will:\n\n```\n1.  Clone the DQN Zoo repository.\n1.  Build a Docker image with all necessary dependencies and run unit tests.\n1.  Start a short run of DQN on Pong in a GPU-accelerated container.\n```\n\n\u003c!-- mdlint on --\u003e\n\nNOTE: `run.sh`, `Dockerfile` and `docker_requirements.txt` together provide a\nself-contained example of the dependencies and commands needed to run an agent\nin DQN Zoo. Using Docker is not a requirement and if `Dockerfile` is not used\nthen the list of dependencies to install may have to be adapted depending on\nyour environment. Also it is not a hard requirement to run on the GPU. Agents\ncan be run on the CPU by specifying the flag `--jax_platform_name=cpu`.\n\n## Goals\n\n*   Serve as a collection of reference implementations of DQN-based agents\n    developed at DeepMind.\n*   Reproduce results reported in papers, on average.\n*   Implement agents purely in Python, using JAX, Haiku and RLax.\n*   Have minimal dependencies.\n*   Be easy to read.\n*   Be easy to modify and customize after forking.\n\n## Non-goals\n\n*   Be a library or framework (these agents are intended to be forked for\n    research).\n*   Be flexible, general and support multiple use cases (at odds with\n    understandability).\n*   Support many environments (users can easily add new ones).\n*   Include every DQN variant that exists.\n*   Incorporate many cool libraries (harder to read, easy for the user to do\n    this after forking, different users prefer different libraries, less\n    self-contained).\n*   Optimize speed and efficiency at the cost of readability or matching\n    algorithmic details in the papers (no C++, keep to a single stream of\n    experience).\n\n## Code structure\n\n*   Each directory contains a published DQN variant configured to run on Atari.\n*   `agent.py` in each agent directory contains an agent class that includes\n    `reset()`, `step()`, `get_state()`, `set_state()` methods.\n*   `parts.py` contains functions and classes used by many of the agents\n    including classes for accumulating statistics and the main training and\n    evaluation loop `run_loop()`.\n*   `replay.py` contains functions and classes relating to experience replay.\n*   `networks.py` contains Haiku networks used by the agents.\n*   `processors.py` contains components for standard Atari preprocessing.\n\n## Implementation notes\n\nGenerally we went with a flatter approach for easier code comprehension.\nExcessive nesting, indirection and generalization have been avoided, but not to\nthe extreme of having a single file per agent. This has resulted in some degree\nof code duplication, but this is less of a maintenance issue as the code base is\nintended to be relatively static.\n\nSome implementation details:\n\n*   The main training and evaluation loop `parts.run_loop()` is implemented as a\n    generator to decouple it from other concerns like logging statistics and\n    checkpointing.\n*   We adopted the pattern of returning a new JAX PRNG key from jitted\n    functions. This allows for splitting keys inside jitted functions which is\n    currently more efficient than splitting outside and passing a key in.\n*   Agent functions to be jitted are defined inline in the agent class\n    `__init__()` instead of as decorated class methods. This emphasizes such\n    functions should be free of side-effects; class methods are generally not\n    pure as they often alter the class instance.\n*   `parts.NullCheckpoint` is a placeholder for users to optionally plug in a\n    checkpointing library appropriate for the file system they are using. This\n    would allow resuming an interrupted training run.\n*   The preprocessing and action repeat logic lives inside each agent. Doing\n    this instead of taking the common approach of environment wrappers allows\n    the run loop to see the \"true\" timesteps. This makes things like recording\n    performance statistics and videos easier since the unmodified rewards and\n    observations are readily available. It also allows us to express all\n    relevant flag values in terms of environment frames, instead of a more\n    confusing mix of environment frames and learning steps.\n\n## Learning curves\n\n\u003c!-- mdlint off() --\u003e\n\nLearning curve data is included in [`results.tar.gz`](results.tar.gz). The\narchive contains a CSV file for each agent, with statistics logged during\ntraining runs. These training runs span the standard set of 57 Atari games, 5\nseeds each, using default agent settings. Note\n[Gym](http://github.com/openai/gym) was used instead of\n[Xitari](http://github.com/deepmind/xitari).\n\n\u003c!-- mdlint on --\u003e\n\nThese CSV files can be theoretically equivalently generated by the following\npseudocode:\n\n```bash\nfor agent in \"${AGENTS[@]}\"; do\n  for game in \"${ATARI_GAMES[@]}\"; do\n    for seed in {1..5}; do\n      python -m \"dqn_zoo.${agent}.run_atari\" \\\n          --environment_name=\"${game}\" \\\n          --seed=\"${seed}\" \\\n          --results_csv_path=\"/tmp/dqn_zoo/${agent}/${game}/${seed}/results.csv\"\n    done\n  done\ndone\n```\n\nEach agent CSV file in `results.tar.gz` is then a concatenation of all\nassociated `results.csv` files, with additional `environment_name` and `seed`\nfields. Note the learning curve data is missing `state_value` since logging for\nthis quantity was added after the data was generated.\n\nPlots show the average score at periodic evaluation phases during training. Each\nepisode during evaluation starts with up to 30 random no-op actions and lasts a\nmaximum of 30 minutes. To make the plots more readable, scores have been\nsmoothed using a moving average with window size 10.\n\nPlot of average score on each individual Atari game for each agent:\n\n![Plot individual](plot_atari_individual.svg)\n\n## FAQ\n\n### Q: Do these agents replicate results from their respective papers?\n\nWe aim to replicate the mean and median human normalized score over all 57 Atari\ngames and to implement the algorithm described in each paper as closely as\npossible.\n\nHowever there are potential sources of differences at the level of an individual\ngame. These include:\n\n*   Differences between [Gym](http://github.com/openai/gym) +\n    [Arcade Learning Environment (ALE)](http://github.com/mgbellemare/Arcade-Learning-Environment)\n    and [Xitari](http://github.com/deepmind/xitari).\n*   Changes in underlying libraries such as the exact image resizing algorithm\n    used in the observation preprocessing.\n*   Atari ROM version.\n\n### Q: Is the execution of these agents deterministic?\n\nWe try to allow for it on CPU. However it is easily broken and note that\nconvolutions on GPU are not deterministic. To allow for determinism we:\n\n*   Build a new environment at the start of every iteration.\n*   Include in the training state:\n    *   Random number generator state.\n    *   Target network parameters (in addition to online network parameters).\n    *   Evaluation agent.\n\n### Q: Why is DQN-based agent X not included?\n\nThere was a bias towards implementing the variants the authors are most familiar\nwith. Also one or more of the following reasons may apply:\n\n*   Did not get round to implementing X.\n*   Have yet to replicate the algorithmic details and learning performance of X.\n*   It is easy to create X from components in DQN Zoo.\n\n### Q: Why not incorporate library / environment X?\n\nX is probably very useful, but every additional library or feature is another\nthing new users need to read and understand. Also everyone differs in the\nauxiliary libraries they like to use. So the recommendation is to fork the agent\nyou want and incorporate the features you wish in the copy. This also gives us\nthe usual benefits of keeping dependencies to a minimum.\n\n### Q: Can I generalize X, then I can do Y with minimal modifications?\n\nCode generalization often makes code harder to read. This is not intended to be\na library in the sense that you import an agent and inject customized components\nto do research. Instead it is designed to be easy to customize after forking. So\nrather than be everything for everyone, we aimed to keep things minimal. Then\nusers can fork and generalize in the directions they specifically care about.\n\n### Q: Why Gym instead of Xitari?\n\nMost DeepMind papers with experiments on Atari published results on\n[Xitari](http://github.com/deepmind/xitari), a fork of the\n[Arcade Learning Environment (ALE)](http://github.com/mgbellemare/Arcade-Learning-Environment).\nThe learning performance of agents in DQN Zoo were also verified on Xitari.\nHowever since [Gym](http://github.com/openai/gym) and the ALE are more widely\nused we have chosen to open source DQN Zoo using Gym. This does introduce\nanother source of differences, though the settings for the Gym Atari\nenvironments have been chosen so they behave as similar as possible to Xitari.\n\n## Contributing\n\nNote we are currently not accepting contributions. See\n[`CONTRIBUTING.md`](CONTRIBUTING.md) for details.\n\n## Citing DQN Zoo\n\nIf you use DQN Zoo in your research, please cite the papers corresponding to the\nagents used and this repository:\n\n```\n@software{dqnzoo2020github,\n  title = {{DQN} {Zoo}: Reference implementations of {DQN}-based agents},\n  author = {John Quan and Georg Ostrovski},\n  url = {http://github.com/deepmind/dqn_zoo},\n  version = {1.2.0},\n  year = {2020},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-deepmind%2Fdqn_zoo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle-deepmind%2Fdqn_zoo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-deepmind%2Fdqn_zoo/lists"}