{"id":22000057,"url":"https://github.com/learnables/cherry","last_synced_at":"2025-04-05T17:09:52.390Z","repository":{"id":35144155,"uuid":"159752575","full_name":"learnables/cherry","owner":"learnables","description":"A PyTorch Library for Reinforcement Learning Research","archived":false,"fork":false,"pushed_at":"2023-06-26T01:44:45.000Z","size":8696,"stargazers_count":198,"open_issues_count":3,"forks_count":31,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-03-29T16:09:49.930Z","etag":null,"topics":["learning","pytorch","reinforcement","reinforcement-learning","rl"],"latest_commit_sha":null,"homepage":"http://cherry-rl.net","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/learnables.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2018-11-30T01:46:38.000Z","updated_at":"2025-03-21T16:07:57.000Z","dependencies_parsed_at":"2023-02-11T07:16:00.073Z","dependency_job_id":null,"html_url":"https://github.com/learnables/cherry","commit_stats":{"total_commits":385,"total_committers":10,"mean_commits":38.5,"dds":"0.038961038961038974","last_synced_commit":"f4164a53dcc762ac5ce53a761fb54f3f69847f90"},"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/learnables%2Fcherry","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/learnables%2Fcherry/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/learnables%2Fcherry/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/learnables%2Fcherry/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/learnables","download_url":"https://codeload.github.com/learnables/cherry/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247369954,"owners_count":20927928,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["learning","pytorch","reinforcement","reinforcement-learning","rl"],"created_at":"2024-11-29T23:09:11.331Z","updated_at":"2025-04-05T17:09:52.372Z","avatar_url":"https://github.com/learnables.png","language":"Python","funding_links":[],"categories":["Reinforcement Learning (RL) and Deep Reinforcement Learning (DRL)"],"sub_categories":["RL/DRL Algorithm Implementations and Software Frameworks"],"readme":"\u003cp align=\"center\"\u003e\u003cimg src=\"http://cherry-rl.net/assets/images/cherry_full.png\" height=\"128px\" /\u003e\u003c/p\u003e\n\n--------------------------------------------------------------------------------\n\n[![Build Status](https://travis-ci.org/learnables/cherry.svg?branch=master)](https://travis-ci.org/learnables/cherry)\n\nCherry is a reinforcement learning framework for researchers built on top of PyTorch.\n\nUnlike other reinforcement learning implementations, cherry doesn't implement a single monolithic  interface to existing algorithms.\nInstead, it provides you with low-level, common tools to write your own algorithms.\nDrawing from the UNIX philosophy, each tool strives to be as independent from the rest of the framework as possible.\nSo if you don't like a specific tool, you don’t need to use it.\n\n**Features**\n\nCherry extends PyTorch with only a handful of new core concepts.\n\n* PyTorch modules for reinforcement learning: \n    * [`cherry.nn.Policy`](http://cherry-rl.net/api/cherry.nn/#cherry.nn.policy.Policy): base class for $\\pi(a \\mid s)$ policies.\n    * [`cherry.nn.ActionValue`](http://cherry-rl.net/api/cherry.nn/#cherry.nn.action_value.ActionValue): base class for $Q(s, a)$ action-value functions.\n* Data structures for reinforcement learning compatible with PyTorch:\n    * [`cherry.Transition`](http://cherry-rl.net/api/cherry/#cherry.experience_replay.Transition): namedtuple to store $(s_t, a_t, r_t, s_{t+1})$ transitions (and more).\n    * [`cherry.ExperienceReplay`](http://cherry-rl.net/api/cherry/#cherry.experience_replay.ExperienceReplay): a list-like buffer to store and sample transitions.\n * Low-level interface *à la* PyTorch to write and debug your algorithms.\n    * [`cherry.td.*`](http://cherry-rl.net/api/cherry.td/) and [`cherry.pg.*`](http://cherry-rl.net/api/cherry.pg/): temporal difference and policy gradient utilities.\n    * [`cherry.algorithms.*`](http://cherry-rl.net/api/cherry.algorithms/): helper functions for popular algorithms ([PPO](http://cherry-rl.net/api/cherry.algorithms/#cherry.algorithms.ppo.PPO), [TD3](http://cherry-rl.net/api/cherry.algorithms/#cherry.algorithms.td3.TD3), [DrQ](http://cherry-rl.net/api/cherry.algorithms/#cherry.algorithms.drq.DrQ), and [more](http://cherry-rl.net/api/cherry.algorithms/)).\n    * [`cherry.debug.*`](http://cherry-rl.net/api/cherry.debug/) and [`cherry.plot.*`](http://cherry-rl.net/api/cherry.plot/): logging, visualization, and debugging tools.\n\nTo learn more about the tools and philosophy behind cherry, check out our [Getting Started tutorial](http://cherry-rl.net/tutorials/getting_started/).\n\n## Overview and Examples\n\nThe following snippet showcases a few of the tools offered by cherry.\nMany more high-quality examples are available in the [examples/](./examples/) folder.\n\n#### Defining a [`cherry.nn.Policy`](http://cherry-rl.net/api/cherry.nn/#cherry.nn.policy.Policy)\n\n~~~python\nclass VisionPolicy(cherry.nn.Policy):  # inherits from torch.nn.Module\n\n   def __init__(self, feature_extractor, actor):\n      super(VisionGaussianPolicy, self).__init__()\n      self.feature_extractor = feature_extractor\n      self.actor = actor\n\n   def forward(self, obs):\n      mean = self.actor(self.feature_extractor(obs))\n      std = 0.1 * torch.ones_like(mean)\n      return cherry.distributions.TanhNormal(mean, std)  # policies always return a distribution\n\npolicy = VisionPolicy(MyResnetExtractor(), MyMLPActor())\naction = policy.act(obs)  # sampled from policy's distribution\ndeterministic_action = policy.act(obs, deterministic=True)  # distribution's mode\naction_distribution = policy(obs)  # work with the policy's distribution\n~~~\n\n#### Building a [`cherry.ExperienceReplay`](http://cherry-rl.net/api/cherry/#cherry.experience_replay.ExperienceReplay) of [`cherry.Transition`](http://cherry-rl.net/api/cherry/#cherry.experience_replay.Transition)\n\n~~~python\n# building the replay\nreplay = cherry.ExperienceReplay()\nstate = env.reset()\nfor t in range(1000):\n   action = policy.act(state)\n   next_state, reward, done, info = env.step(action)\n   replay.append(state, action, reward, next_state, done)\n   next_state = state\n\n# manipulating the replay\nreplay = replay[-256:]  # indexes like a list\nbatch = replay.sample(32, contiguous=True)  # sample transitions into a replay\nbatch = batch.to('cuda') # move replay to device\nfor transition in reversed(batch): # iterate over a replay\n   transition.reward *= 0.99\n\n# get all states, actions, and rewards as PyTorch tensors.\nreinforce_loss = - torch.sum(policy(batch.state()).log_prob(batch.action()) * batch.reward())\n~~~\n\n#### Designing algorithms with [`cherry.td`](http://cherry-rl.net/api/cherry.td/), [`cherry.pg`](http://cherry-rl.net/api/cherry.pg/), and [`cherry.algorithms`](http://cherry-rl.net/api/cherry.algorithms/)\n\n~~~python\n# defining a new algorithm\n@dataclasses.dataclass\nclass MyA2C:\n   discount: float = 0.99\n   \n   def update(self, replay, policy, state_value, optimizer):\n      # discount rewards\n      values = state_value(replay.action())\n      discounted_rewards = cherry.td.discount(\n         self.discount, replay.reward(), replay.done(), bootstrap=values[-1].detach()\n      )\n\n      # Compute losses\n      policy_loss = cherry.algorithms.A2C.policy_loss(\n         log_probs=policy(replay.state()).log_prob(replay.action()),\n         advantages=discounted_rewards - values.detach(),\n      )\n      value_loss = cherry.algorithms.A2C.state_value_loss(values, discounted_rewards)\n\n      # Optimization step\n      optimizer.zero_grad()\n      (policy_loss + value_loss).backward()\n      optimizer.step()\n      return {'a2c/policy_loss': policy_loss, 'a2c/value_loss': value_loss}\n\n# using MyA2C\nmy_a2c = MyA2C(discount=0.95)\nmy_policy = MyPolicy()\nlinear_value = cherry.models.LinearValue(128)\nadam = torch.optim.Adam(policy.parameters())\nfor step in range(1000):\n   replay = collect_experience(policy)\n   my_a2c.update(replay, my_policy, linear_value, adam)\n~~~\n\n## Install\n\n```\npip install cherry-rl\n```\n\n## Changelog\n\nA human-readable changelog is available in the [CHANGELOG.md](CHANGELOG.md) file.\n\n## Documentation\n\nDocumentation and tutorials are available on cherry’s website: [http://cherry-rl.net](http://cherry-rl.net).\n\n## Contributing\n\nHere are a couple of guidelines we strive to follow.\n\n* It's always a good idea to open an issue first, where we can discuss how to best proceed.\n* If you want to contribute a new example using cherry, it would preferably stand in a single file.\n* If you would like to contribute a new feature to the core library, we suggest to first implement an example showcasing your new functionality. Doing so is quite useful:\n    * it allows for automatic testing,\n    * it ensures that the functionality is correctly implemented,\n    * it shows users how to use your functionality, and\n    * it gives a concrete example when discussing the best way to merge your implementation.\n\n## Acknowledgements\n\nCherry draws inspiration from many reinforcement learning implementations, including\n\n* [OpenAI Baselines](https://github.com/openai/baselines),\n* John Schulman's [implementations](https://github.com/joschu/modular_rl)\n* Ilya Kostrikov's [implementations](https://github.com/ikostrikov/pytorch-a2c-ppo-acktr),\n* Shangtong Zhang's [implementations](https://github.com/ShangtongZhang/DeepRL),\n* Dave Abel's [implementations](https://github.com/david-abel/simple_rl/),\n* Vitchyr Pong's [implementations](https://github.com/vitchyr/rlkit),\n* Kai Arulkumaran's [implementations](https://github.com/Kaixhin/spinning-up-basic),\n* [RLLab](https://github.com/rll/rllab) / [Garage](https://github.com/rlworkgroup/garage).\n\n\n## Why 'cherry' ?\n\nBecause it's the sweetest part of [the cake](https://twitter.com/ylecun/status/1097532314614034433).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flearnables%2Fcherry","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flearnables%2Fcherry","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flearnables%2Fcherry/lists"}