{"id":23117932,"url":"https://github.com/bourbonut/dqn-pacman","last_synced_at":"2026-04-22T23:34:58.592Z","repository":{"id":40553161,"uuid":"485912905","full_name":"bourbonut/dqn-pacman","owner":"bourbonut","description":"Deep Q-Network on the Atari Game Ms-Pacman","archived":false,"fork":false,"pushed_at":"2024-05-23T09:26:29.000Z","size":6377,"stargazers_count":8,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-27T21:38:07.194Z","etag":null,"topics":["atari","dqn","pacman","reinforcement-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bourbonut.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-26T18:59:48.000Z","updated_at":"2025-03-21T16:10:51.000Z","dependencies_parsed_at":"2022-09-10T14:11:36.048Z","dependency_job_id":"018c2278-7550-4d03-b6e5-e140fea87f35","html_url":"https://github.com/bourbonut/dqn-pacman","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/bourbonut/dqn-pacman","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bourbonut%2Fdqn-pacman","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bourbonut%2Fdqn-pacman/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bourbonut%2Fdqn-pacman/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bourbonut%2Fdqn-pacman/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bourbonut","download_url":"https://codeload.github.com/bourbonut/dqn-pacman/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bourbonut%2Fdqn-pacman/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32159959,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-22T17:06:48.269Z","status":"ssl_error","status_checked_at":"2026-04-22T17:06:19.037Z","response_time":58,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["atari","dqn","pacman","reinforcement-learning"],"created_at":"2024-12-17T04:31:56.194Z","updated_at":"2026-04-22T23:34:58.560Z","avatar_url":"https://github.com/bourbonut.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Deep Q-Network on the Atari Game Ms-Pacman\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./docs/demo.gif\"/\u003e\n\u003c/p\u003e\n\n## Basic information\n\nThe goal of this project is to apply the **Deep Q-Network** algorithm on Ms-Pacman environment and to reach good performance without using prioritized replay memory or better DQN namely _A3C_ or _Rainbow DQN_.\nFollowing results were obtained with following parameters :\n\n| Parameter          | Value               |\n| ------------------ | ------------------- |\n| Batch size         | `128`               |\n| Discount rate      | `0.99`              |\n| Epsilon max        | `1.0`               |\n| Epsilon min        | `0.1`               |\n| Epsilon decay      | `1,000,000`         |\n| Target update      | `8,000`             |\n| Replay memory size | `18,000`            |\n| Optimizer          | SGD with momentum   |\n| Learning rate      | `2.5e-4`            |\n| Momentum           | `0.95`              |\n| Positive reward    | `log(reward, 1000)` |\n| Negative reward    | `-log(20, 1000)`    |\n\n## Performances\n\n### Rewards\n\nDefault rewards follow a large range. To standardize rewards, the logarithm function is applied on reward given by the environment (see the function `transform_reward` in `utils/utils.py`).\n\n```python\nfrom math import log\ndef transform_reward(reward):\n    return log(reward, 1000) if reward \u003e 0 else reward\n```\n\nAlso a negative reward is given to the agent when a ghost eats the agent. On the top of the following figure, the average reward is computed on the 20 latest episodes :\n\n```python\nimport statistics\ndef mov_avg(self, t):\n    # t = 20\n    values = (\n\t[0] * (t - len(self._total)) + self._total\n\tif len(self._total) \u003c t\n\telse self._total[-t:]\n    )\n    self._mean.append(statistics.mean(values))\n```\n\n![rewards](./docs/rewards.png)\n\n### Q-value\n\nThe behavior of the agent becomes better when the Q-value improves over the time.\n\n![q-value](./docs/q_values.png)\n\n### Results\n\n#### Smart behavior\n\nThe agent is able to avoid chasing ghosts\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./docs/good_behavior.gif\"/\u003e\n\u003c/p\u003e\n\n#### High score\n\nThe agent eats pills and eats ghosts after having gotten a boost. \n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./docs/high_score.gif\"/\u003e\n\u003c/p\u003e\n\n## For installation\n\nIt is **highly recommended** to install packages in a virtual environment.\n\n### Installation of Atari environment\n\n```sh\npip install ale-py==0.7\nwget http://www.atarimania.com/roms/Roms.rar\nunrar e Roms.rar\nunzip -qq ROMS.zip\nale-import-roms /content/ROMS/ | grep pacman\n\npip install -U gym\npip install -U gym[atari]\n```\n\n### Installation of dependencies\n\n```sh\npip install -r requirements.txt\n```\n\n**Note :** If you don't follow the requirements file, `opencv-python` and `matplotlib` could be incompatible depending on the versions of packages. `opencv-python` is only used to write a video in `eval.py`.\n\n## For usage\n\n### Training part\n\n#### Train the agent\n\nIn `deep_Q_network` folder, you can find the file `parameters.py` where parameters are set. After checking them, you can run the training with the following command line\n\n```sh\npython main.py\n```\n\n#### Train and save evolution step by step (a lot of memory)\n\nTo save the evolution step by step, simply run:\n```sh\npython main.py --image\n```\n\n![example-result](./docs/example-result.png)\n\n#### Dynamic display\n\nThis mode is useful when you want to see how the agent reacts and interacts with its environment.\n\nTo display the \"dashboard\", simply run :\n```sh\npython main.py --stream\n```\nThen enter the URL `localhost:5000` in your browser.\n\n![dashboard](./docs/board.png)\n\n**Note :**: It is recommended for a long training to not use this mode.\n\n### Evaluation\n\n#### Location of saved data\n\nWhen you run `main.py`, it will automatically create a folder `results` in where all results will be stored.\n\n#### Usage\n\nBy default, the evaluation from `eval.py` is on the most recent **folder** and **episode**.\nTo specify them :\n```sh\npython eval.py -e 120 --path ./results/mytrainingfolder\n```\nYou can find different flags to get what you want :\n- by default, it saves a plot with Q values, rewards and the last losses of desired episode.\n- `--reward`, it saves rewards with a pseudo moving average\n- `--qvalue`, it saves Q values with a pseudo moving average\n- `--record`, it records the agent interaction\n- `-a` or `--all`, it records the agent interaction and save plots\n\n\n## Structure of the code\n```\n.\n├── deep_Q_network\n│   ├── __init__.py\n│   ├── buffer.py # buffer class used for websocket and for tracking training performances\n│   ├── memory.py # replay memory\n│   ├── model.py # dueling DQN and optimization (see the class for more details)\n│   ├── parameters.py # all parameters except how rewards are managed\n│   └── preprocessing.py # for preprocessing observations\n├── docs\n│   └── ...\n├── evaluation # only use by `eval.py`\n│   ├── __init__.py\n│   ├── parser.py\n│   └── utils.py\n├── utils\n│   ├── __init__.py\n│   ├── actions.py\n│   ├── opencv.py\n│   ├── parser.py\n│   ├── path.py\n│   ├── rewards.py\n│   └── save_functions.py\n├── results\n│   └── training-[...]\n│       ├── models # folder with pytorch models\n│       │   ├── policy-model-[...].pt\n│       │   └── target-model-[...].pt\n│       ├── plots # folder for `python main.py --image` command\n│       │   └── episode-[...].png\n│       ├── recorded-data # folder with pickle files\n│       │   └── episode-[...].pkl\n│       ├── output_video.avi\n│       ├── q_values.png\n│       ├── result.png\n│       └── rewards.png\n├── eval.py # to evaluate the agent\n├── main.py # to train the agent\n├── README.md\n└── requirements.txt\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbourbonut%2Fdqn-pacman","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbourbonut%2Fdqn-pacman","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbourbonut%2Fdqn-pacman/lists"}