{"id":19023509,"url":"https://github.com/neymarl/pacman-rl","last_synced_at":"2025-07-11T06:39:42.152Z","repository":{"id":109047656,"uuid":"150361864","full_name":"NeymarL/Pacman-RL","owner":"NeymarL","description":"Implement some reinforcement learning algorithms, test and visualize on Pacman.","archived":false,"fork":false,"pushed_at":"2018-12-03T07:40:15.000Z","size":7608,"stargazers_count":27,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-23T09:40:58.976Z","etag":null,"topics":["actor-critic","pacman","policy","policy-gradient","q-learning","reinforcement-learning","sarsa-lambda"],"latest_commit_sha":null,"homepage":"https://www.52coding.com.cn/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NeymarL.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-09-26T03:12:34.000Z","updated_at":"2025-03-28T02:08:15.000Z","dependencies_parsed_at":"2023-04-05T23:33:09.326Z","dependency_job_id":null,"html_url":"https://github.com/NeymarL/Pacman-RL","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/NeymarL/Pacman-RL","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NeymarL%2FPacman-RL","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NeymarL%2FPacman-RL/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NeymarL%2FPacman-RL/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NeymarL%2FPacman-RL/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NeymarL","download_url":"https://codeload.github.com/NeymarL/Pacman-RL/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NeymarL%2FPacman-RL/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264749869,"owners_count":23658288,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["actor-critic","pacman","policy","policy-gradient","q-learning","reinforcement-learning","sarsa-lambda"],"created_at":"2024-11-08T20:30:18.274Z","updated_at":"2025-07-11T06:39:42.096Z","avatar_url":"https://github.com/NeymarL.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Pacman-RL\n\nImplement some reinforcement learning algorithms, test and visualize on Pacman under [OpenAI's Gym](https://gym.openai.com/) environment.\n\n## Requirements\n\n* Python 3.6+\n* gym\n* matplotlib\n* tensorflow\n* keras\n* mujoco_py (if you want to save replay)\n* torch\n* torchvision\n\n## Run\n\n* Run `python run.py --controller MC train` for training using Monte-Carlo control. The weight file will be saved as `weights/mc.h5`.\n* Run `python run.py --controller MC --render --show_plot --evaluate_episodes 10 evaluate` for evaluation using Monte-Carlo control. It will render the Pacman environment and show the dynamic Q-value and reward plot at the same time.\n\n```\nFull usage: run.py [-h]\n              [--controller {MC,Sarsa,Sarsa_lambda,Q_learning,REINFORCE,ActorCritic,A3C,PPO}]\n              [--render] [--save_replay] [--save_plot] [--show_plot]\n              [--num_episodes NUM_EPISODES] [--batch_size BATCH_SIZE]\n              [--eva_interval EVA_INTERVAL]\n              [--evaluate_episodes EVALUATE_EPISODES] [--lr LR]\n              [--epsilon EPSILON] [--gamma GAMMA] [--lam LAM] [--forward]\n              [--max_workers MAX_WORKERS] [--t_max T_MAX]\n              {train,evaluate}\n\npositional arguments:\n  {train,evaluate}      what to do\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --controller {MC,Sarsa,Sarsa_lambda,Q_learning,REINFORCE,ActorCritic,A3C,PPO}\n                        choose an algorithm (controller)\n  --render              set to render the env when evaluate\n  --save_replay         set to save replay\n  --save_plot           set to save Q-value plot when evaluate\n  --show_plot           set to show Q-value plot when evaluate\n  --num_episodes NUM_EPISODES\n                        set to run how many episodes\n  --batch_size BATCH_SIZE\n                        set the batch size\n  --eva_interval EVA_INTERVAL\n                        set how many episodes evaluate once\n  --evaluate_episodes EVALUATE_EPISODES\n                        set evaluate how many episodes\n  --lr LR               set learning rate\n  --epsilon EPSILON     set epsilon when use epsilon-greedy\n  --gamma GAMMA         set reward decay rate\n  --lam LAM             set lambda if use sarsa(lambda) algorithm\n  --forward             set to use forward-view sarsa(lambda)\n  --rawpixels           set to use raw pixels as input (only valid to PPO)\n  --max_workers MAX_WORKERS\n                        set max workers to train\n  --t_max T_MAX         set simulate how many timesteps until update param\n```\n\n![sample1](graph/sample1.gif)\n\n![sample2](graph/sample2.gif)\n\n## Reinforcement Learning Algorithms\n\n### Monte-Carlo Control\n\n* Policy evaluation\n    * ![](http://latex.codecogs.com/gif.latex?Q%28s_t%2C%20a_t%29%20%5Cleftarrow%20Q%28s_t%2C%20a_t%29%20\u0026plus;%20%5Cfrac%7B1%7D%7BN%28s_t%2C%20a_t%29%7D%28G_t%20-%20Q%28s_t%2C%20a_t%29%29)\n    * ![](http://latex.codecogs.com/gif.latex?G_t%20%3D%20R_%7Bt%20\u0026plus;%201%7D%20\u0026plus;%20%5Cgamma%20R_%7Bt\u0026plus;2%7D%20\u0026plus;%20...%20\u0026plus;%20%5Cgamma%5E%7BT-1%7DR_T)\n\n* Policy improvement: 𝜀-greedy with 𝜀 decay\n* Q-value function approximation: A fully connected layer (input layer and output layer with no hidden layer)\n\n![learning curve](graph/mc/mc.png)\n\n### Sarsa(0)\n\n* Policy evaluation\n    * ![](http://latex.codecogs.com/gif.latex?Q%28s%2Ca%29%5Cleftarrow%20Q%28s%2Ca%29\u0026plus;%5Calpha%28R\u0026plus;%5Cgamma%20Q%28s%27%2Ca%27%29-Q%28s%2Ca%29%29)\n* Policy improvement: 𝜀-greedy with 𝜀 decay\n* Q-value function approximation: A fully connected layer (input layer and output layer with no hidden layer)\n\n![learning curve](graph/sarsa/sarsa.png)\n\n### Sarsa(𝝀)\n\n**Forward-view**\n\n* Policy evaluation\n    * ![](http://latex.codecogs.com/gif.latex?Q%28s%2C%20a%29%20%5Cleftarrow%20Q%28s%2C%20a%29%20\u0026plus;%20%5Calpha%28q_t%5E%5Clambda-Q%28s%2Ca%29%29)\n    * ![](http://latex.codecogs.com/gif.latex?q_t%5E%5Clambda%3D%281-%5Clambda%29%5Csum_%7Bn%3D1%7D%5E%5Cinfty%20%5Clambda%5E%7Bn-1%7Dq_t%5E%7B%28n%29%7D)\n    * ![](http://latex.codecogs.com/gif.latex?q_t%5E%7B%28n%29%7D%3DR_%7Bt\u0026plus;1%7D%20\u0026plus;%20%5Cgamma%20R_%7Bt\u0026plus;2%7D%20\u0026plus;%20...%20\u0026plus;%20%5Cgamma%5E%7Bn-1%7D%20R_%7Bt\u0026plus;n%7D\u0026plus;%5Cgamma%5En%20Q%28s_%7Bt\u0026plus;n%7D%2C%20a_%7Bt\u0026plus;n%7D%29)\n* Policy improvement: 𝜀-greedy with 𝜀 decay\n* Q-value function approximation: A fully connected layer (input layer and output layer with no hidden layer)\n\n**Backward-view**\n\n* Policy evaluation\n    * ![](http://latex.codecogs.com/gif.latex?Q%28s%2C%20a%29%20%5Cleftarrow%20Q%28s%2C%20a%29%20\u0026plus;%20%5Calpha%28R%20\u0026plus;%20%5Cgamma%20Q%28s%27%2C%20a%27%29%20-%20Q%28s%2C%20a%29%29*E_t%28s%2C%20a%29)\n    * Accumulating eligibility trace: ![](http://latex.codecogs.com/gif.latex?E_t%28s%2C%20a%29%20%3D%5Cgamma%5Clambda%20E_%7Bt-1%7D%28s%2C%20a%29%20\u0026plus;%201%28S_t%20%3D%20s%2C%20A_t%20%3D%20a%29)\n* Policy improvement: 𝜀-greedy with 𝜀 decay\n* Q-value function approximation: A fully connected layer (input layer and output layer with no hidden layer)\n\n![learning curve](graph/sarsa_lambda/sarsa_lambda.png)\n\n### Q-learning\n\n* Policy evaluation\n    * ![](http://latex.codecogs.com/gif.latex?Q%28s%2Ca%29%5Cleftarrow%20Q%28s%2C%20a%29\u0026plus;%5Calpha%28R%20\u0026plus;%20%5Cgamma%20%5Cmax_%7Ba%27%7DQ%28s%27%2C%20a%27%29%20-%20Q%28s%2C%20a%29%29)\n* Policy improvement: 𝜀-greedy with 𝜀 decay\n* Q-value function approximation: A fully connected layer (input layer and output layer with no hidden layer)\n\n![learning curve](graph/q_learning/q_learning.png)\n\n### REINFORCE\n\n**Monte-Carlo policy gradient**\n\n* Use return Gt to estimate ![](http://latex.codecogs.com/gif.latex?Q_%5Cpi%28s%2C%20a%29): ![](http://latex.codecogs.com/gif.latex?%5Ctheta%20%5Cleftarrow%20%5Ctheta%20\u0026plus;%20%5Calpha%20%5Ctriangledown_%5Ctheta%20%5Clog%20%5Cpi_%5Ctheta%28s_t%2C%20a_t%29G_t)\n* Policy function approximation: Softmax policy with a fc layer\n\n**Note**: You shold pick a very small `lr` to train a decent model, e.g. `lr = 0.00001`\n![learning curve](graph/reinforce/reinforce.png)\n\n### Advantage Actor-Critic\n\n* Actor\n  * Softmax policy with a fc layer\n  * Use advantage function to estimate ![](http://latex.codecogs.com/gif.latex?Q_%5Cpi%28s%2C%20a%29): ![](http://latex.codecogs.com/gif.latex?%5Ctriangledown_%5Ctheta%20J%28%5Ctheta%29%20%3D%20E_%5Cpi%5B%5Ctriangledown_%5Ctheta%5Clog%5Cpi%28s%2C%20a%29A_%5Cpi%28s%2C%20a%29%29%5D%20%3D%20E_%5Cpi%5B%5Ctriangledown_%5Ctheta%5Clog%5Cpi%28s%2C%20a%29%5Cdelta_%5Cpi%5D), where ![](http://latex.codecogs.com/gif.latex?A_%5Cpi%28s%2C%20a%29%20%3D%20Q_%5Cpi%28s%2C%20a%29%20-%20V_%5Cpi%28s%29)\n\n* Critic\n  * TD policy evaluation ![](http://latex.codecogs.com/gif.latex?V_%5Cpi%28s%29%20%5Cleftarrow%20V_%5Cpi%28s%29%20\u0026plus;%20%5Calpha%5Cdelta_%5Cpi)\n  * ![](http://latex.codecogs.com/gif.latex?%5Cdelta_%5Cpi%20%3D%20R%20\u0026plus;%20%5Cgamma%20V_%5Cpi%28s%27%29%20-%20V_%5Cpi%28s%29)\n  * Value function approximation: a fully connected layer (input layer and output layer with no hidden layer)\n\n![learning curve](graph/actorcritic/actorcritic.png)\n\n### Asynchronous Advantage Actor-Critic (A3C)\n\n![a3c](graph/a3c/algorithm.jpeg)\n\n![a3c](graph/a3c/a3c.png)\n\n### Trust Region Policy Optimization (TRPO)\n\n![trpo](graph/trpo.png)\n**Note**: Running with OpenAI [Spinning Up](https://github.com/openai/spinningup), TRPO is not implemented in this repo.\n\n### Proximal Policy Optimization (PPO)\n\n![algo](graph/ppo/ppo_algo.png)\n\nRun with:\n\n```bash\npython run.py --controller PPO --max_worker 6 --gamma 0.99 --evaluate_episodes 50 --batch_size 20 --epsilon 0.2 --lam 0.97 --eva_interval 100 train\n```\n\n![ppo](graph/ppo/ppo.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneymarl%2Fpacman-rl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fneymarl%2Fpacman-rl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneymarl%2Fpacman-rl/lists"}