{"id":21135272,"url":"https://github.com/cloneofsimo/policy-optimization-torch","last_synced_at":"2026-04-27T17:02:58.889Z","repository":{"id":44927031,"uuid":"510673628","full_name":"cloneofsimo/policy-optimization-torch","owner":"cloneofsimo","description":null,"archived":false,"fork":false,"pushed_at":"2022-07-16T06:24:43.000Z","size":2253,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-21T06:43:24.260Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cloneofsimo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-07-05T09:40:52.000Z","updated_at":"2022-07-06T15:24:41.000Z","dependencies_parsed_at":"2022-08-04T01:00:19.863Z","dependency_job_id":null,"html_url":"https://github.com/cloneofsimo/policy-optimization-torch","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloneofsimo%2Fpolicy-optimization-torch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloneofsimo%2Fpolicy-optimization-torch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloneofsimo%2Fpolicy-optimization-torch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloneofsimo%2Fpolicy-optimization-torch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cloneofsimo","download_url":"https://codeload.github.com/cloneofsimo/policy-optimization-torch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243581061,"owners_count":20314163,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-20T06:48:01.303Z","updated_at":"2025-12-29T17:18:29.093Z","avatar_url":"https://github.com/cloneofsimo.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Policy Optimization implemented with PyTorch\n\n\nBenchmarking VPG \u0026 PPO with various weights ($\\Psi$) for the policy gradient term, where $\\Psi$ is used as:\n\n$$\\begin{align*}\ng_\\theta := \u0026 \\nabla J_\\theta (\\pi_\\theta) \\\\\n=\u0026 \\mathbb{E}\\big[ \\nabla_\\theta \\sum \\log \\pi_\\theta(a_t|s_t) R \\big] \\\\\n=\u0026 \\mathbb{E}\\big[ \\sum_{t = 0} ^ \\infty  \\Psi_t \\nabla_\\theta  \\log \\pi_\\theta(a_t | s_t ) \\big]\n\\end{align*}$$\n\n\n# How does this differ from Spinning Up RL's Implementation?\n\nThis repo is heavily based on [Spinning Up RL's](https://spinningup.openai.com/en/latest/index.html) implementation of VPG and PPO.\n\nHowever there are some changes. First, similar to various other object-oriented pipelines such as [Timm](https://github.com/rwightman/pytorch-image-models), [Avalenche](https://avalanche.continualai.org/), etc., we have used pytorch \u0026 pythonic OOP structured training for these policy optimization algorithms.\n\nWe have also removed some of the features that are not essential to the algorithm themselves, and on the other hand, we have added some new features that are not present in the original implementations.\n\n# Results\n\n![](contents/ppoVSvpg.png)\n\nWhere $\\Psi$ is one of:\n\n1. Discounted return:\n\n$$\n\\Psi_t = \\sum_{l = 0} ^ \\infty \\gamma^l r_l\n$$\n\n2. Reward-to-go:\n\n$$\n\\Psi_t = \\sum_{l = 0} ^ \\infty \\gamma^{t + l} r_{t + l}\n$$\n\n3. Reward-to-go with baseline:\n\n$$\n\\Psi_t = \\sum_{l = 0} ^ \\infty \\gamma^{t + l} r_{t + l} - b(s_t)\n$$\n\nWhere we have used baseline as $b(s_t) = V(s_t)$.\n\n4. Discounted Temporal Difference Residual:\n\n$$\n\\Psi_t = \\delta_t := r_t + \\gamma V(s_{t+1}) - V(s_t)\n$$\n\n5. Generalized Advantage Estimation:\n\n$$\n\\Psi_t = \\sum_{l = 0} ^ \\infty (\\gamma \\lambda)^l \\delta_{t + l}\n$$\n\nWhere $\\delta_{t}$ is the discounted TD residual defined on 4.\n\n## Note\nUsing the original definition from the paper proposing [GAE](https://arxiv.org/abs/1506.02438), $\\lambda\u003e0$ \nhas the same value as 5. By their definition, setting $\\lambda=0$\nyields 4. But we can see that this is not the case if we evauate using 5. It's not a big deal, because this shenanigan happend since\n${0}^{0}$ is not well defined. Follow equation (16) from the paper for more details.\n\n# Updates!\n\nFaster buffer, cuda support, checked working on continuous environment(gym's BipedalWalker-v3)\n\nFixed minor bugs.\n\n![](contents/conti-ppo.png)\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloneofsimo%2Fpolicy-optimization-torch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcloneofsimo%2Fpolicy-optimization-torch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloneofsimo%2Fpolicy-optimization-torch/lists"}