{"id":16739366,"url":"https://github.com/zafarali/policy-gradient-methods","last_synced_at":"2025-09-07T01:05:12.254Z","repository":{"id":73849184,"uuid":"118181156","full_name":"zafarali/policy-gradient-methods","owner":"zafarali","description":"Modular PyTorch implementation of policy gradient methods","archived":false,"fork":false,"pushed_at":"2018-11-15T21:45:48.000Z","size":83,"stargazers_count":25,"open_issues_count":4,"forks_count":6,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-04-10T13:34:15.028Z","etag":null,"topics":["policy-gradient","reinforcement-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zafarali.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-01-19T21:36:14.000Z","updated_at":"2022-11-02T15:10:07.000Z","dependencies_parsed_at":null,"dependency_job_id":"582a0cce-91bd-4458-b71a-dc255bf9caf1","html_url":"https://github.com/zafarali/policy-gradient-methods","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/zafarali/policy-gradient-methods","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zafarali%2Fpolicy-gradient-methods","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zafarali%2Fpolicy-gradient-methods/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zafarali%2Fpolicy-gradient-methods/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zafarali%2Fpolicy-gradient-methods/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zafarali","download_url":"https://codeload.github.com/zafarali/policy-gradient-methods/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zafarali%2Fpolicy-gradient-methods/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273983110,"owners_count":25202095,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-06T02:00:13.247Z","response_time":2576,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["policy-gradient","reinforcement-learning"],"created_at":"2024-10-13T00:50:33.605Z","updated_at":"2025-09-07T01:05:12.225Z","avatar_url":"https://github.com/zafarali.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Policy Gradient Methods\n\nPyTorch implementation of policy gradient methods.\n\n*NOTE* This repository is still work in progress! As I continue to try to break things down into modular and reusable parts things might break. However, I will try to ensure the cases in tests keep passing.\n\n## Installation\n\nThis library only works with Python 3.5+. If you are using Python 2.7 you should upgrade immediately. \nThe requirements for this library can be found in `requirements.txt`. To install this library you can use pip:\n\n```bash\npip install -e .\n```\n\nThe `-e` indicates that the library will be installed in development mode. You can then check if it works by opening up python and typing:\n\n```python\nimport pg_methods\nprint(pg_methods.__version__) # should print 0\n```\n\n## Tests\n\nThere are tests for components in this library under `./tests/`. You can run them by executing `python -m pytest ./tests --verbose`.\n\n## Philosophy\n\nThere are a few [good](https://github.com/vitchyr/rlkit) [reinforcement learning](https://github.com/ikostrikov/pytorch-a2c-ppo-acktr) [reinforcement](https://github.com/ikostrikov/pytorch-ddpg-naf) [algorithm](https://github.com/jingweiz/pytorch-rl) algorithm implementations in pytorch. \nThere are many ones in [Tensorflow](https://github.com/openai/baselines), [Theano](https://github.com/rll/rllab) and [Keras](https://github.com/keras-rl/keras-rl).\nThe main thing lacking in the PyTorch implementations is extensibility/modularity. Sure I would love to run this one algorithm on all environments ever. But sometimes it's just the little parts that are useful. For example,\na good utility to calculate [discounted future returns with masks](https://github.com/zafarali/policy-gradient-methods/blob/f7fc0b2ba06c208f97a73f1b9f0bdd0507fd2f54/pg_methods/gradients.py#L5-L24).\nOr the [REINFORCE objective](https://github.com/zafarali/policy-gradient-methods/blob/f7fc0b2ba06c208f97a73f1b9f0bdd0507fd2f54/pg_methods/objectives/objectives.py#L40-L74) itself. \nMaybe you want to try a new kind of [baseline](https://github.com/zafarali/policy-gradient-methods/blob/master/pg_methods/baselines.py)? The goal of this library is to allow you to do all of these things. \nSort of like LEGO. Arguably, more important than having a long script with the algorithm, is having the components to make new ones. This is one thing I find frustrating with baselines, all the algorithms are in their own folders, with only marginal code sharing. \nI've already used some stuff from here in [some](https://github.com/zafarali/deep-subsets/blob/master/experiments/integer_version_set2subset_RL.py) (old version of `pg_methods`) of [my projects](https://github.com/zafarali/better-sampling/blob/master/rvi_sampling/samplers/RVISampler.py) (soon to be released).\n\nTo see how the code is organized see [./pg_methods/README.md](./pg_methods/README.md)\n\n## Algorithms Implemented\n\n1. Vanilla Policy Gradient (`pg_methods.algorithms.VanillaPolicyGradient`)\n\n### To be implemented (contributions welcome!)\n\n- [ ] Synchronous Advantage Actor Critic\n- [ ] Asynchronous Advantage Actor Critic\n- [ ] Natural Policy Gradient\n- [ ] Trust Region Policy Optimization\n- [ ] Proximal Policy Optimization\n\netc.\n\n## other opportunities to contribute\n\nSee [projects](https://github.com/zafarali/policy-gradient-methods/projects). Things like new `objectives`, `baselines` `optimizers`, `replay_memory`s are all good contributions!\n\nAlso what would be cool is a large scale benchmarking script so that we can run all the algorithms to see how they perform on different gym environments.\n\n### Some performance graphs (soon to improve)\n\n![rewards-plots](https://user-images.githubusercontent.com/6295292/37562262-a87510b0-2a3a-11e8-95bf-e3b8799bd546.png)\n\nI'm working to get `roboschool` installed on the ComputeCanada clusters so i can run for longer. To install roboschool on your local machine you can [try this script](https://gist.github.com/zafarali/7186f8790c8c4288a9c4e72c04f8c8ce)\n\n![rewards-roboschool](https://user-images.githubusercontent.com/6295292/37562263-aa016ea6-2a3a-11e8-95cc-beb68a4caa42.png)\n\n## Example\n\nHere is an example script of how to get started with the `VanillaPolicyGradient` algorithm. We expect other algorithms to have similar interfaces.\n\n```python\nfrom pg_methods import interfaces\nfrom pg_methods.algorithms.REINFORCE import VanillaPolicyGradient\nfrom pg_methods.baselines import FunctionApproximatorBaseline\nfrom pg_methods.utils import experiment\n\nenv = interfaces.make_parallelized_gym_env('CartPole-v0', seed=4, n_workers=2)\nexperiment_logger = experiment.Experiment({'algorithm_name': 'VPG'}, './')\nexperiment_logger.start()\nfn_approximator, policy = experiment.setup_policy(env, hidden_non_linearity=nn.ReLU, hidden_sizes=[16, 16])\noptimizer = torch.optim.SGD(fn_approximator.parameters(), lr=0.01)\n\n# setting up a baseline function\nbaseline_approximator = MLP_factory(env.observation_space_info['shape'][0],\n                               [16, 16],\n                               output_size=1,\n                               hidden_non_linearity=nn.ReLU)\nbaseline_optimizer = torch.optim.SGD(baseline_approximator.parameters(), lr=0.01)\nbaseline = FunctionApproximatorBaseline(baseline_approximator, baseline_optimizer)\n\nalgorithm = VanillaPolicyGradient(env, policy, optimizer, gamma=0.99, baseline=baseline)\n\nrewards, losses = algorithm.run(1000, verbose=True)\n\nexperiment_logger.log_data('rewards', rewards.tolist())\nexperiment_logger.save()\n```    \n\nMore example scripts can be seen in [./experiments/](./experiments)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzafarali%2Fpolicy-gradient-methods","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzafarali%2Fpolicy-gradient-methods","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzafarali%2Fpolicy-gradient-methods/lists"}