{"id":21197634,"url":"https://github.com/jason-cky/deeprl-pytorch","last_synced_at":"2025-07-10T05:31:54.247Z","repository":{"id":65070013,"uuid":"294942060","full_name":"Jason-CKY/DeepRL-pytorch","owner":"Jason-CKY","description":"Pytorch implementations of various Deep Reinforcement Learning algorithms on pybullet environments.","archived":false,"fork":false,"pushed_at":"2022-02-18T16:46:26.000Z","size":270011,"stargazers_count":29,"open_issues_count":2,"forks_count":6,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-04T07:06:04.511Z","etag":null,"topics":["ddpg","ppo","pybullet-environments","python3","pytorch-implementation","reinforcement-learning-algorithms","rlbench","td3","trpo"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Jason-CKY.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-09-12T12:44:30.000Z","updated_at":"2025-04-10T09:46:42.000Z","dependencies_parsed_at":"2023-01-13T15:24:51.536Z","dependency_job_id":null,"html_url":"https://github.com/Jason-CKY/DeepRL-pytorch","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Jason-CKY/DeepRL-pytorch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jason-CKY%2FDeepRL-pytorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jason-CKY%2FDeepRL-pytorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jason-CKY%2FDeepRL-pytorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jason-CKY%2FDeepRL-pytorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Jason-CKY","download_url":"https://codeload.github.com/Jason-CKY/DeepRL-pytorch/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jason-CKY%2FDeepRL-pytorch/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264535993,"owners_count":23624404,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ddpg","ppo","pybullet-environments","python3","pytorch-implementation","reinforcement-learning-algorithms","rlbench","td3","trpo"],"created_at":"2024-11-20T19:45:47.946Z","updated_at":"2025-07-10T05:31:49.209Z","avatar_url":"https://github.com/Jason-CKY.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Project Deprecated\n\nPlease note that there will not be any updates to this project in the foreseeable future. Please do not add any issues to this repo expecting a fix or explanation. Some of the libraries have had breaking updates (gym) and my `requirements.txt` did not state the version requirements and its pretty much impossible to reproduce the experiments now. However, there is value in looking at the implementation of the various RL algorithms.\n\nPlease consider forking this project if you want to continue working on it and provide support with newer environments and libraries.\n\n# Deep RL policies on Pybullet Environments\n\nThis repo is a pytorch implementation of various deep RL algorithms, trained and evaluated on pybullet robotic environments.\n\n## Dependencies:\n* CUDA \u003e= 10.2\n* [RLBench](https://github.com/stepjam/RLBench)\n\n## Implemented Algorithms:\n\n\u003ctable\u003e\n    \u003cthead\u003e\n        \u003ctr\u003e\n            \u003cth\u003eName\u003c/th\u003e\n            \u003cth\u003eDiscrete actions\u003c/th\u003e\n            \u003cth\u003eContinuous actions\u003c/th\u003e\n            \u003cth\u003eStochastic policy\u003c/th\u003e\n            \u003cth\u003eDeterministic policy\u003c/th\u003e\n        \u003c/tr\u003e\n    \u003c/thead\u003e\n    \u003ctbody\u003e\n        \u003ctr\u003e\n            \u003ctd\u003e DDPG \u003c/td\u003e\n            \u003ctd\u003e :x: \u003c/td\u003e\n            \u003ctd\u003e :heavy_check_mark: \u003c/td\u003e\n            \u003ctd\u003e :x: \u003c/td\u003e\n            \u003ctd\u003e :heavy_check_mark: \u003c/td\u003e\n        \u003c/tr\u003e\n        \u003ctr\u003e\n            \u003ctd\u003e TD3 \u003c/td\u003e\n            \u003ctd\u003e :x: \u003c/td\u003e\n            \u003ctd\u003e :heavy_check_mark: \u003c/td\u003e\n            \u003ctd\u003e :x: \u003c/td\u003e\n            \u003ctd\u003e :heavy_check_mark: \u003c/td\u003e\n        \u003c/tr\u003e\n         \u003ctr\u003e\n            \u003ctd\u003e TRPO \u003c/td\u003e\n            \u003ctd\u003e :heavy_check_mark: \u003c/td\u003e\n            \u003ctd\u003e :heavy_check_mark: \u003c/td\u003e\n            \u003ctd\u003e :heavy_check_mark: \u003c/td\u003e\n            \u003ctd\u003e :x: \u003c/td\u003e\n        \u003c/tr\u003e\n        \u003ctr\u003e\n            \u003ctd\u003e PPO \u003c/td\u003e\n            \u003ctd\u003e :heavy_check_mark: \u003c/td\u003e\n            \u003ctd\u003e :heavy_check_mark: \u003c/td\u003e\n            \u003ctd\u003e :heavy_check_mark: \u003c/td\u003e\n            \u003ctd\u003e :x: \u003c/td\u003e\n        \u003c/tr\u003e       \n        \u003ctr\u003e\n            \u003ctd\u003e Option-Critic \u003c/td\u003e\n            \u003ctd\u003e :heavy_check_mark: \u003c/td\u003e\n            \u003ctd\u003e :heavy_check_mark: \u003c/td\u003e\n            \u003ctd\u003e :heavy_check_mark: \u003c/td\u003e\n            \u003ctd\u003e :x: \u003c/td\u003e\n        \u003c/tr\u003e   \n        \u003ctr\u003e\n            \u003ctd\u003e DAC_PPO \u003c/td\u003e\n            \u003ctd\u003e :heavy_check_mark: \u003c/td\u003e\n            \u003ctd\u003e :heavy_check_mark: \u003c/td\u003e\n            \u003ctd\u003e :heavy_check_mark: \u003c/td\u003e\n            \u003ctd\u003e :x: \u003c/td\u003e\n        \u003c/tr\u003e   \n    \u003c/tbody\u003e\n\u003c/table\u003e\n\n## Environments Supported\nThe following gym environments are supported on this repo.\n* OpenAI gym's environments\n* Pybullet gym environments\n* RLBench gym environments\n\n## Types of Networks Implemented:\n* Multi-Layered Perceptron (MLP)\n* Convolutional Neural Network (CNN)\n* Variational Autoencoders (VAE)\n\n* hidden_sizes are the number of neurons in each of the dense layer of the MLP.\n* conv_layer_sizes is a list containing the parameters of each convolutional layer, i.e. [output_channel, kernel_size, stride]\n\nTo use mlp neural net, set ac_kwargs['model_type'] to 'mlp'\n\n```\n\"ac_kwargs\": {\n    \"model_type\": \"mlp\"\n    \"hidden_sizes\": [256, 256]\n}\n```\n\nTo use cnn neural net, set ac_kwargs['model_type'] to 'cnn'\n\n```\n\"ac_kwargs\": {\n    \"model_type\": \"cnn\"\n    \"hidden_sizes\": [512, 256],\n    \"conv_layer_sizes\": [[16, 5, 2],\n    [32, 5, 2], \n    [64, 5, 2], \n    [64, 3, 1]]\n}\n```\n\nTo use cnn neural net, set ac_kwargs['model_type'] to 'vae'. \n```\n\"ac_kwargs\": {\n    \"model_type\": \"vae\",\n    \"vae_weights_path\": \"VAE/output/vae_reach_target-vision-v0_wrist_rgb.pth\",\n    \"hidden_sizes\": [512, 256]\n}\n```\n\n## VAE network\nVAE network needs to be pretrained on the environment's images before being used on the RL algorithm. The data generation and training code are provided at [VAE directory](VAE/README.md)\n\n## Comparison of results in PyBullet Environments\n\u003ctable\u003e\n    \u003cthead\u003e\n        \u003ctr\u003e\n            \u003cth\u003eEnvironment\u003c/th\u003e\n            \u003cth\u003e Learning Curve \u003c/th\u003e\n            \u003cth\u003e Episode Recording \u003c/th\u003e\n        \u003c/tr\u003e\n    \u003c/thead\u003e\n    \u003ctbody\u003e\n        \u003ctr\u003e\n            \u003ctd\u003e CartPole Continuous BulletEnv-v0 \u003c/td\u003e\n            \u003ctd\u003e \u003cimg src = 'Model_Weights\\CartPoleContinuousBulletEnv-v0\\comparison.png'\u003e \u003c/td\u003e\n            \u003ctd\u003e\u003cimg src = 'Model_Weights\\CartPoleContinuousBulletEnv-v0\\ddpg\\recording.gif'\u003e \u003c/td\u003e\n        \u003c/tr\u003e\n        \u003ctr\u003e\n            \u003ctd\u003e Hopper BulletEnv-v0 \u003c/td\u003e\n            \u003ctd\u003e \u003cimg src = 'Model_Weights\\HopperBulletEnv-v0\\comparison.png'\u003e \u003c/td\u003e\n            \u003ctd\u003e\u003cimg src = 'Model_Weights\\HopperBulletEnv-v0\\td3\\recording.gif'\u003e \u003c/td\u003e\n        \u003c/tr\u003e\n        \u003ctr\u003e\n            \u003ctd\u003e AntBulletEnv-v0 \u003c/td\u003e\n            \u003ctd\u003e \u003cimg src = 'Model_Weights\\AntBulletEnv-v0\\comparison.png'\u003e \u003c/td\u003e\n            \u003ctd\u003e\u003cimg src = 'Model_Weights\\AntBulletEnv-v0\\td3\\recording.gif'\u003e \u003c/td\u003e\n        \u003c/tr\u003e\n        \u003ctr\u003e\n            \u003ctd\u003e HalfCheetahBulletEnv-v0 \u003c/td\u003e\n            \u003ctd\u003e \u003cimg src = 'Model_Weights\\HalfCheetahBulletEnv-v0\\comparison.png'\u003e \u003c/td\u003e\n            \u003ctd\u003e\u003cimg src = 'Model_Weights\\HalfCheetahBulletEnv-v0\\ddpg\\recording.gif'\u003e \u003c/td\u003e\n        \u003c/tr\u003e\n    \u003c/tbody\u003e\n\u003c/table\u003e\n\n## Results of Option-Critic on RLBench Environments\nThe agents are trained on the front-rgb camera view to solve the RLBench Manipulation Tasks.\n\u003ctable\u003e\n    \u003cthead\u003e\n        \u003ctr\u003e\n            \u003cth\u003eEnvironment\u003c/th\u003e\n            \u003cth\u003e Learning Curve \u003c/th\u003e\n            \u003cth\u003e Episode Recording \u003c/th\u003e\n        \u003c/tr\u003e\n    \u003c/thead\u003e\n    \u003ctbody\u003e\n        \u003ctr\u003e\n            \u003ctd\u003e open-box \u003c/td\u003e\n            \u003ctd\u003e \u003cimg src = 'Model_Weights\\open_box-vision-v0\\comparison_full.png'\u003e \u003c/td\u003e\n            \u003ctd\u003e\u003cimg src = 'Model_Weights\\open_box-vision-v0\\oc_conv\\recording.gif'\u003e \u003c/td\u003e\n        \u003c/tr\u003e\n        \u003ctr\u003e\n            \u003ctd\u003e close-box \u003c/td\u003e\n            \u003ctd\u003e \u003cimg src = 'Model_Weights\\close_box-vision-v0\\comparison_full.png'\u003e \u003c/td\u003e\n            \u003ctd\u003e\u003cimg src = 'Model_Weights\\close_box-vision-v0\\oc_vae\\recording.gif'\u003e \u003c/td\u003e\n        \u003c/tr\u003e\n    \u003c/tbody\u003e\n\u003c/table\u003e\n\n## How to use\n* Clone this repo\n* pip install -r requirements.txt\n\n### Training model for openai gym environment\n* Edit training parameters in ./Algorithms/\u003calgo\u003e/\u003calgo\u003e_config.json\n```\npython train.py\nusage: train.py [-h] [--env ENV] [--agent {ddpg,trpo,ppo,td3,random}]\n                [--arch {mlp,cnn}] --timesteps TIMESTEPS [--seed SEED]\n                [--num_trials NUM_TRIALS] [--normalize] [--rlbench] [--image]\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --env ENV             environment_id\n  --agent {ddpg,trpo,ppo,td3,random}\n                        specify type of agent\n  --arch {mlp,cnn}      specify architecture of neural net\n  --timesteps TIMESTEPS\n                        specify number of timesteps to train for\n  --seed SEED           seed number for reproducibility\n  --num_trials NUM_TRIALS\n                        Number of times to train the algo\n  --normalize           if true, normalize environment observations\n  --rlbench             if true, use rlbench environment wrappers\n  --image               if true, use rlbench environment wrappers\n```\n\n### Testing trained model performance\n```\npython test.py\nusage: test.py [-h] [--env ENV] [--agent {ddpg,trpo,ppo,td3,random}]\n               [--arch {mlp,cnn}] [--render] [--gif] [--timesteps TIMESTEPS]\n               [--seed SEED] [--normalize] [--rlbench] [--image]\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --env ENV             environment_id\n  --agent {ddpg,trpo,ppo,td3,random}\n                        specify type of agent\n  --arch {mlp,cnn}      specify architecture of neural net\n  --render              if true, display human renders of the environment\n  --gif                 if true, make gif of the trained agent\n  --timesteps TIMESTEPS\n                        specify number of timesteps to train for\n  --seed SEED           seed number for reproducibility\n  --normalize           if true, normalize environment observations\n  --rlbench             if true, use rlbench environment wrappers\n  --image               if true, use rlbench environment wrappers\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjason-cky%2Fdeeprl-pytorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjason-cky%2Fdeeprl-pytorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjason-cky%2Fdeeprl-pytorch/lists"}