{"id":20430640,"url":"https://github.com/jihoonerd/deep-reinforcement-learning-with-double-q-learning","last_synced_at":"2025-04-12T20:33:40.648Z","repository":{"id":37202887,"uuid":"222670120","full_name":"jihoonerd/Deep-Reinforcement-Learning-with-Double-Q-learning","owner":"jihoonerd","description":"📖 Paper: Deep Reinforcement Learning with Double Q-learning 🕹️","archived":false,"fork":false,"pushed_at":"2024-05-09T15:27:10.000Z","size":19602,"stargazers_count":51,"open_issues_count":0,"forks_count":16,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-26T14:50:25.467Z","etag":null,"topics":["atari","ddqn","deep-learning","deepmind","dqn","reinforcement-learning","tensorflow2"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jihoonerd.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-11-19T10:29:59.000Z","updated_at":"2025-03-21T16:08:57.000Z","dependencies_parsed_at":"2023-01-17T14:02:02.663Z","dependency_job_id":"1200e3a5-6eb3-4589-82b7-2413700026ca","html_url":"https://github.com/jihoonerd/Deep-Reinforcement-Learning-with-Double-Q-learning","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jihoonerd%2FDeep-Reinforcement-Learning-with-Double-Q-learning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jihoonerd%2FDeep-Reinforcement-Learning-with-Double-Q-learning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jihoonerd%2FDeep-Reinforcement-Learning-with-Double-Q-learning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jihoonerd%2FDeep-Reinforcement-Learning-with-Double-Q-learning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jihoonerd","download_url":"https://codeload.github.com/jihoonerd/Deep-Reinforcement-Learning-with-Double-Q-learning/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248629809,"owners_count":21136321,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["atari","ddqn","deep-learning","deepmind","dqn","reinforcement-learning","tensorflow2"],"created_at":"2024-11-15T08:08:08.315Z","updated_at":"2025-04-12T20:33:40.591Z","avatar_url":"https://github.com/jihoonerd.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Deep Reinforcement Learning with Double Q-learning\n\n![atlantis_playing](/assets/atlantis.gif)\n\nThis repository implements the paper: **[Deep Reinforcement Learning with Double Q-learning](https://arxiv.org/abs/1509.06461)**.\n\nThe authors of the paper applied [Double Q-learning](https://papers.nips.cc/paper/3964-double-q-learning) concept on their DQN algorithm. This paper proposed Double DQN, which is similar to DQN but more robust to overestimation of Q-values.\n\nThe major difference between those two algorithms is the way to calculate Q-value from target network. Compared to the DQN, directly using Q-value from target network, DDQN chooses an action that maximizes the Q-value of main network at the next state.\n\n#### DQN\n![dqn_y_target](/assets/y_dqn.png)\n\n#### DDQN\n![ddqn_y_target](/assets/y_ddqn.png)\n\nMost of the implementation is almost the same as the [implementation of DQN](https://github.com/jihoonerd/Human-level-control-through-deep-reinforcement-learning).\n\n## Features\n\n* Employed ***TensorFlow 2*** with performance optimization\n* Simple structure\n* Easy to reproduce\n\n## Model Structure\n\n![nn.svg](/assets/nn.svg)\n\n## Requirements\n\n***Default running environment is assumed to be CPU-ONLY. If you want to run this repo on GPU machine, just replace `tensorflow` to `tensorflow-gpu` in package lists.***\n\n## How to install\n\n### `virtualenv`\n\n```bash\n$ virtualenv venv\n$ source venv/bin/activate\n$ pip install -r requirements.txt\n```\n\n## How to run\n\nYou can run Atari 2600 game with `main.py`. Running environment needs to be `NoFrameskip` from `gym` package.\n\n```bash\n$ python main.py --help\nusage: main.py [-h] [--env ENV] [--train] [--play PLAY]\n               [--log_interval LOG_INTERVAL]\n               [--save_weight_interval SAVE_WEIGHT_INTERVAL]\n\nAtari: DQN\noptional arguments:\n  -h, --help            show this help message and exit\n  --env ENV             Should be NoFrameskip environment\n  --train               Train agent with given environment\n  --play PLAY           Play with a given weight directory\n  --log_interval LOG_INTERVAL\n                        Interval of logging stdout\n  --save_weight_interval SAVE_WEIGHT_INTERVAL\n                        Interval of saving weights\n```\n\n### Example 1: Train BreakoutNoFrameskip-v4\n\n``` bash\n$ python main.py --env BreakoutNoFrameskip-v4 --train\n```\n\n### Example 2: Play PongNoFrameskip-v4 with trained weights\n\n```bash\n$ python main.py --env PongNoFrameskip-v4 --play ./log/[LOGDIR]/weights\n```\n\n### Example 3: Control log \u0026 save interval\n\n```bash\n$ python main.py --env BreakoutNoFrameskip-v4 --train --log_interval 100 --save_weight_interval 1000\n```\n\n## Results\n\nThis implementation is guaranteed to work well for `Atlantis`, `Boxing`, `Breakout` and `Pong`. Tensorboard summary is located at `./archive`. Tensorboard will show following information:\n\n* Average Q value\n* Epsilon (for exploration)\n* Latest 100 avg reward (clipped)\n* Loss\n* Reward (clipped)\n* Test score\n* Total frames\n\n```bash\n$ tensorboard --logdir=./archive/\n```\n\nSingle RTX 2080 Ti is used for the results below. (Thanks to [@JKeun](https://github.com/JKeun) for allowing his computation resources)\n\n### Atalntis\n\n* Orange: DQN\n* Blue: DDQN\n\n#### Reward\n\n![atlantis](/assets/atlantis_result.png)\n\n#### Q-value\n\n![atlantis_Q](/assets/DDQN_Q-value.png)\n\nWe can see that DDQN's average Q-value is suppressed compared to that of DQN.\n\n## BibTeX\n\n```\n@article{hasselt2015doubledqn,\n  abstract = {The popular Q-learning algorithm is known to overestimate action values under\ncertain conditions. It was not previously known whether, in practice, such\noverestimations are common, whether they harm performance, and whether they can\ngenerally be prevented. In this paper, we answer all these questions\naffirmatively. In particular, we first show that the recent DQN algorithm,\nwhich combines Q-learning with a deep neural network, suffers from substantial\noverestimations in some games in the Atari 2600 domain. We then show that the\nidea behind the Double Q-learning algorithm, which was introduced in a tabular\nsetting, can be generalized to work with large-scale function approximation. We\npropose a specific adaptation to the DQN algorithm and show that the resulting\nalgorithm not only reduces the observed overestimations, as hypothesized, but\nthat this also leads to much better performance on several games.},\n  added-at = {2019-11-18T11:40:13.000+0100},\n  author = {van Hasselt, Hado and Guez, Arthur and Silver, David},\n  biburl = {https://www.bibsonomy.org/bibtex/2c2bad4b4c5a34cb31a3f569c71e851ab/jan.hofmann1},\n  description = {[1509.06461] Deep Reinforcement Learning with Double Q-learning},\n  interhash = {d3061c37961afb78096e314854dd90bc},\n  intrahash = {c2bad4b4c5a34cb31a3f569c71e851ab},\n  keywords = {dqn q-learning reinforcement_learning},\n  note = {cite arxiv:1509.06461Comment: AAAI 2016},\n  timestamp = {2019-11-18T11:40:13.000+0100},\n  title = {Deep Reinforcement Learning with Double Q-learning},\n  url = {http://arxiv.org/abs/1509.06461},\n  year = 2015\n}\n```\n\n## Author\nJihoon Kim ([@jihoonerd](https://github.com/jihoonerd))\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjihoonerd%2Fdeep-reinforcement-learning-with-double-q-learning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjihoonerd%2Fdeep-reinforcement-learning-with-double-q-learning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjihoonerd%2Fdeep-reinforcement-learning-with-double-q-learning/lists"}