{"id":18551363,"url":"https://github.com/morvanzhou/pytorch-a3c","last_synced_at":"2025-04-04T14:05:24.389Z","repository":{"id":108437943,"uuid":"117970516","full_name":"MorvanZhou/pytorch-A3C","owner":"MorvanZhou","description":"Simple A3C implementation with pytorch + multiprocessing","archived":false,"fork":false,"pushed_at":"2023-03-10T07:28:08.000Z","size":142,"stargazers_count":632,"open_issues_count":17,"forks_count":146,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-03-28T13:06:07.245Z","etag":null,"topics":["a3c","actor-critic","asynchronous-advantage-actor-critic","gym","multiprocessing","neural-network","pytorch","toy-example"],"latest_commit_sha":null,"homepage":"https://mofanpy.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MorvanZhou.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-01-18T10:51:29.000Z","updated_at":"2025-03-23T23:30:15.000Z","dependencies_parsed_at":"2023-06-01T02:30:39.577Z","dependency_job_id":null,"html_url":"https://github.com/MorvanZhou/pytorch-A3C","commit_stats":{"total_commits":16,"total_committers":3,"mean_commits":5.333333333333333,"dds":0.125,"last_synced_commit":"5ab27abee2c3ac3ca921ac393bfcbda4e0a91745"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MorvanZhou%2Fpytorch-A3C","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MorvanZhou%2Fpytorch-A3C/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MorvanZhou%2Fpytorch-A3C/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MorvanZhou%2Fpytorch-A3C/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MorvanZhou","download_url":"https://codeload.github.com/MorvanZhou/pytorch-A3C/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247189685,"owners_count":20898692,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["a3c","actor-critic","asynchronous-advantage-actor-critic","gym","multiprocessing","neural-network","pytorch","toy-example"],"created_at":"2024-11-06T21:08:49.930Z","updated_at":"2025-04-04T14:05:24.364Z","avatar_url":"https://github.com/MorvanZhou.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Simple implementation of Reinforcement Learning (A3C) using Pytorch\n\nThis is a toy example of using multiprocessing in Python to asynchronously train a\nneural network to play discrete action [CartPole](https://gym.openai.com/envs/CartPole-v0/) and\ncontinuous action [Pendulum](https://gym.openai.com/envs/Pendulum-v0/) games.\nThe asynchronous algorithm I used is called [Asynchronous Advantage Actor-Critic](https://arxiv.org/pdf/1602.01783.pdf) or A3C.\n\nI believe it would be the simplest toy implementation you can find at the moment (2018-01).\n\n## What are the main focuses in this implementation?\n\n* Pytorch + multiprocessing (NOT threading) for parallel training\n* Both discrete and continuous action environments\n* To be simple and easy to dig into the code (less than 200 lines)\n\n## Reason of using [Pytorch](http://pytorch.org/) instead of [Tensorflow](https://www.tensorflow.org/)\n\nBoth of them are great for building your customized neural network. But to work\nwith multiprocessing, Tensorflow is not that great due to its low compatibility with multiprocessing.\nI have an implementation of [Tensorflow A3C build on threading](https://github.com/MorvanZhou/Reinforcement-learning-with-tensorflow/tree/master/contents/10_A3C).\nI even tried to implement [distributed Tensorflow](https://github.com/MorvanZhou/Reinforcement-learning-with-tensorflow/blob/master/contents/10_A3C/A3C_distributed_tf.py).\nHowever, the distributed version is for cluster computing which I don't have.\nWhen using only one machine, it is slower than threading version I wrote.\n\nFortunately, Pytorch gets the [multiprocessing compatibility](http://pytorch.org/docs/master/notes/multiprocessing.html).\nI went through many Pytorch A3C examples ([there](https://github.com/ikostrikov/pytorch-a3c), [there](https://github.com/jingweiz/pytorch-rl)\nand [there](https://github.com/ShangtongZhang/DeepRL)). They are great but too complicated to dig into the code.\nTherefore, this is my motivation to write my simple example codes.\n\nBTW, if you are interested to learn Pytorch, [there](https://github.com/MorvanZhou/PyTorch-Tutorial)\n is my simple tutorial code with many visualizations. I also made the tensorflow tutorial (same as pytorch) available in [here](https://github.com/MorvanZhou/Tensorflow-Tutorial).\n\n## Codes \u0026 Results\n\n* [shared_adam.py](/shared_adam.py): optimizer that shares its parameters in parallel\n* [utils.py](/utils.py): useful function that can be used more than once\n* [discrete_A3C.py](/discrete_A3C.py): CartPole, neural net and training for discrete action space\n* [continuous_A3C.py](/continuous_A3C.py): Pendulum, neural net and training for continuous action space\n\nCartPole result\n![cartpole](/results/cartpole.png)\n\nPendulum result\n![pendulum](/results/pendulum.png)\n\n## Dependencies\n\n* pytorch \u003e= 0.4.0\n* numpy\n* gym\n* matplotlib\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmorvanzhou%2Fpytorch-a3c","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmorvanzhou%2Fpytorch-a3c","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmorvanzhou%2Fpytorch-a3c/lists"}