{"id":13936582,"url":"https://github.com/jingweiz/pytorch-rl","last_synced_at":"2025-07-19T22:31:16.442Z","repository":{"id":118625785,"uuid":"87801329","full_name":"jingweiz/pytorch-rl","owner":"jingweiz","description":"Deep Reinforcement Learning with pytorch \u0026 visdom","archived":false,"fork":false,"pushed_at":"2020-07-16T20:01:31.000Z","size":12699,"stargazers_count":797,"open_issues_count":6,"forks_count":144,"subscribers_count":25,"default_branch":"master","last_synced_at":"2024-08-08T23:24:02.283Z","etag":null,"topics":["a3c","acer","actor-critic","deep-learning","deep-reinforcement-learning","dqn","pytorch","pytorch-a3c","reinforcement-learning","trpo","visdom"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jingweiz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-04-10T11:05:53.000Z","updated_at":"2024-08-04T14:09:21.000Z","dependencies_parsed_at":null,"dependency_job_id":"4f8a1a87-056e-4dfe-95bf-a161f10fc403","html_url":"https://github.com/jingweiz/pytorch-rl","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jingweiz%2Fpytorch-rl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jingweiz%2Fpytorch-rl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jingweiz%2Fpytorch-rl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jingweiz%2Fpytorch-rl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jingweiz","download_url":"https://codeload.github.com/jingweiz/pytorch-rl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226686729,"owners_count":17666928,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["a3c","acer","actor-critic","deep-learning","deep-reinforcement-learning","dqn","pytorch","pytorch-a3c","reinforcement-learning","trpo","visdom"],"created_at":"2024-08-07T23:02:48.492Z","updated_at":"2024-11-27T04:31:15.320Z","avatar_url":"https://github.com/jingweiz.png","language":"Python","funding_links":[],"categories":["Python","Paper implementations｜论文实现","Paper implementations"],"sub_categories":["Other libraries｜其他库:","Other libraries:"],"readme":"# **Deep Reinforcement Learning** with\n# **pytorch** \u0026 **visdom**\n*******\n\n\n* Sample testings of trained agents (DQN on Breakout, A3C on Pong, DoubleDQN on CartPole, continuous A3C on InvertedPendulum(MuJoCo)):\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e\u003cimg src=\"/assets/breakout.gif?raw=true\" width=\"200\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"/assets/a3c_pong.gif?raw=true\" width=\"200\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"/assets/cartpole.gif?raw=true\" width=\"200\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"/assets/a3c_con.gif?raw=true\" width=\"200\"\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n* Sample on-line plotting while training an A3C agent on Pong (with 16 learner processes):\n![a3c_pong_plot](/assets/a3c_pong.png)\n\n* Sample loggings while training a DQN agent on CartPole (we use ```WARNING``` as the logging level currently to get rid of the ```INFO``` printouts from visdom):\n```bash\n[WARNING ] (MainProcess) \u003c===================================\u003e\n[WARNING ] (MainProcess) bash$: python -m visdom.server\n[WARNING ] (MainProcess) http://localhost:8097/env/daim_17040900\n[WARNING ] (MainProcess) \u003c===================================\u003e DQN\n[WARNING ] (MainProcess) \u003c-----------------------------------\u003e Env\n[WARNING ] (MainProcess) Creating {gym | CartPole-v0} w/ Seed: 123\n[INFO    ] (MainProcess) Making new env: CartPole-v0\n[WARNING ] (MainProcess) Action Space: [0, 1]\n[WARNING ] (MainProcess) State  Space: 4\n[WARNING ] (MainProcess) \u003c-----------------------------------\u003e Model\n[WARNING ] (MainProcess) MlpModel (\n  (fc1): Linear (4 -\u003e 16)\n  (rl1): ReLU ()\n  (fc2): Linear (16 -\u003e 16)\n  (rl2): ReLU ()\n  (fc3): Linear (16 -\u003e 16)\n  (rl3): ReLU ()\n  (fc4): Linear (16 -\u003e 2)\n)\n[WARNING ] (MainProcess) No Pretrained Model. Will Train From Scratch.\n[WARNING ] (MainProcess) \u003c===================================\u003e Training ...\n[WARNING ] (MainProcess) Validation Data @ Step: 501\n[WARNING ] (MainProcess) Start  Training @ Step: 501\n[WARNING ] (MainProcess) Reporting       @ Step: 2500 | Elapsed Time: 5.32397913933\n[WARNING ] (MainProcess) Training Stats:   epsilon:          0.972\n[WARNING ] (MainProcess) Training Stats:   total_reward:     2500.0\n[WARNING ] (MainProcess) Training Stats:   avg_reward:       21.7391304348\n[WARNING ] (MainProcess) Training Stats:   nepisodes:        115\n[WARNING ] (MainProcess) Training Stats:   nepisodes_solved: 114\n[WARNING ] (MainProcess) Training Stats:   repisodes_solved: 0.991304347826\n[WARNING ] (MainProcess) Evaluating      @ Step: 2500\n[WARNING ] (MainProcess) Iteration: 2500; v_avg: 1.73136949539\n[WARNING ] (MainProcess) Iteration: 2500; tderr_avg: 0.0964358523488\n[WARNING ] (MainProcess) Iteration: 2500; steps_avg: 9.34579439252\n[WARNING ] (MainProcess) Iteration: 2500; steps_std: 0.798395631184\n[WARNING ] (MainProcess) Iteration: 2500; reward_avg: 9.34579439252\n[WARNING ] (MainProcess) Iteration: 2500; reward_std: 0.798395631184\n[WARNING ] (MainProcess) Iteration: 2500; nepisodes: 107\n[WARNING ] (MainProcess) Iteration: 2500; nepisodes_solved: 106\n[WARNING ] (MainProcess) Iteration: 2500; repisodes_solved: 0.990654205607\n[WARNING ] (MainProcess) Saving Model    @ Step: 2500: /home/zhang/ws/17_ws/pytorch-rl/models/daim_17040900.pth ...\n[WARNING ] (MainProcess) Saved  Model    @ Step: 2500: /home/zhang/ws/17_ws/pytorch-rl/models/daim_17040900.pth.\n[WARNING ] (MainProcess) Resume Training @ Step: 2500\n...\n```\n*******\n\n\n## What is included?\nThis repo currently contains the following agents:\n\n- Deep Q Learning (DQN) [[1]](http://arxiv.org/abs/1312.5602), [[2]](http://home.uchicago.edu/~arij/journalclub/papers/2015_Mnih_et_al.pdf)\n- Double DQN [[3]](http://arxiv.org/abs/1509.06461)\n- Dueling network DQN (Dueling DQN) [[4]](https://arxiv.org/abs/1511.06581)\n- Asynchronous Advantage Actor-Critic (A3C) (w/ both discrete/continuous action space support) [[5]](https://arxiv.org/abs/1602.01783), [[6]](https://arxiv.org/abs/1506.02438)\n- Sample Efficient Actor-Critic with Experience Replay (ACER) (currently w/ discrete action space support (Truncated Importance Sampling, 1st Order TRPO)) [[7]](https://arxiv.org/abs/1611.01224), [[8]](https://arxiv.org/abs/1606.02647)\n\nWork in progress:\n- Testing ACER\n\nFuture Plans:\n- Deep Deterministic Policy Gradient (DDPG) [[9]](http://arxiv.org/abs/1509.02971), [[10]](http://proceedings.mlr.press/v32/silver14.pdf)\n- Continuous DQN (CDQN or NAF) [[11]](http://arxiv.org/abs/1603.00748)\n\n\n## Code structure \u0026 Naming conventions:\nNOTE: we follow the exact code structure as [pytorch-dnc](https://github.com/jingweiz/pytorch-dnc) so as to make the code easily transplantable.\n* ```./utils/factory.py```\n\u003e We suggest the users refer to ```./utils/factory.py```,\n where we list all the integrated ```Env```, ```Model```,\n ```Memory```, ```Agent``` into ```Dict```'s.\n All of those four core classes are implemented in ```./core/```.\n The factory pattern in ```./utils/factory.py``` makes the code super clean,\n as no matter what type of ```Agent``` you want to train,\n or which type of ```Env``` you want to train on,\n all you need to do is to simply modify some parameters in ```./utils/options.py```,\n then the ```./main.py``` will do it all (NOTE: this ```./main.py``` file never needs to be modified).\n* namings\n\u003e To make the code more clean and readable, we name the variables using the following pattern (mainly in inherited ```Agent```'s):\n\u003e * ```*_vb```: ```torch.autograd.Variable```'s or a list of such objects\n\u003e * ```*_ts```: ```torch.Tensor```'s or a list of such objects\n\u003e * otherwise: normal python datatypes\n\n\n## Dependencies\n- Python 2.7\n- [PyTorch \u003e=v0.2.0](http://pytorch.org/)\n- [Visdom](https://github.com/facebookresearch/visdom)\n- [OpenAI Gym \u003e=v0.9.0 (for lower versoins, just need to change into the available games, e.g. change PongDeterministic-v4 to PongDeterministic-v3)](https://github.com/openai/gym)\n- [mujoco-py (Optional: for training continuous version of a3c)](https://github.com/openai/mujoco-py)\n*******\n\n\n## How to run:\nYou only need to modify some parameters in ```./utils/options.py``` to train a new configuration.\n\n* Configure your training in ```./utils/options.py```:\n\u003e * ```line 14```: add an entry into ```CONFIGS``` to define your training (```agent_type```, ```env_type```, ```game```, ```model_type```, ```memory_type```)\n\u003e * ```line 33```: choose the entry you just added\n\u003e * ```line 29-30```: fill in your machine/cluster ID (```MACHINE```) and timestamp (```TIMESTAMP```) to define your training signature (```MACHINE_TIMESTAMP```),\n the corresponding model file and the log file of this training will be saved under this signature (```./models/MACHINE_TIMESTAMP.pth``` \u0026 ```./logs/MACHINE_TIMESTAMP.log``` respectively).\n Also the visdom visualization will be displayed under this signature (first activate the visdom server by type in bash: ```python -m visdom.server \u0026```, then open this address in your browser: ```http://localhost:8097/env/MACHINE_TIMESTAMP```)\n\u003e * ```line 32```: to train a model, set ```mode=1``` (training visualization will be under ```http://localhost:8097/env/MACHINE_TIMESTAMP```); to test the model of this current training, all you need to do is to set ```mode=2``` (testing visualization will be under ```http://localhost:8097/env/MACHINE_TIMESTAMP_test```).\n\n* Run:\n\u003e ```python main.py```\n*******\n\n\n## Bonus Scripts :)\nWe also provide 2 additional scripts for quickly evaluating your results after training. (Dependecies: [lmj-plot](https://github.com/lmjohns3/py-plot))\n* ```plot.sh``` (e.g., plot from log file: ```logs/machine1_17080801.log```)\n\u003e * ```./plot.sh machine1 17080801```\n\u003e * the generated figures will be saved into ```figs/machine1_17080801/```\n* ```plot_compare.sh``` (e.g., compare log files: ```logs/machine1_17080801.log```,```logs/machine2_17080802.log```)\n\u003e ```./plot.sh 00 machine1 17080801 machine2 17080802```\n\u003e * the generated figures will be saved into ```figs/compare_00/```\n\u003e * the color coding will be in the order of: ```red green blue magenta yellow cyan```\n*******\n\n\n## Repos we referred to during the development of this repo:\n* [matthiasplappert/keras-rl](https://github.com/matthiasplappert/keras-rl)\n* [transedward/pytorch-dqn](https://github.com/transedward/pytorch-dqn)\n* [ikostrikov/pytorch-a3c](https://github.com/ikostrikov/pytorch-a3c)\n* [onlytailei/A3C-PyTorch](https://github.com/onlytailei/A3C-PyTorch)\n* [Kaixhin/ACER](https://github.com/Kaixhin/ACER)\n* And a private implementation of A3C from [@stokasto](https://github.com/stokasto)\n*******\n\n\n## Citation\nIf you find this library useful and would like to cite it, the following would be appropriate:\n```\n@misc{pytorch-rl,\n  author = {Zhang, Jingwei and Tai, Lei},\n  title = {jingweiz/pytorch-rl},\n  url = {https://github.com/jingweiz/pytorch-rl},\n  year = {2017}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjingweiz%2Fpytorch-rl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjingweiz%2Fpytorch-rl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjingweiz%2Fpytorch-rl/lists"}