{"id":20271659,"url":"https://github.com/ymd-h/cpprb","last_synced_at":"2025-04-04T12:06:19.511Z","repository":{"id":43602656,"uuid":"187312605","full_name":"ymd-h/cpprb","owner":"ymd-h","description":"Fast Flexible Replay Buffer Library (Mirror repository of https://gitlab.com/ymd_h/cpprb)","archived":false,"fork":false,"pushed_at":"2024-12-14T14:23:17.000Z","size":2472,"stargazers_count":72,"open_issues_count":0,"forks_count":10,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-03-28T11:11:54.096Z","etag":null,"topics":["cython","machine-learning","python3","reinforcement-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ymd-h.png","metadata":{"files":{"readme":"README.org","changelog":"CHANGELOG.org","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-05-18T04:08:00.000Z","updated_at":"2024-12-14T14:18:10.000Z","dependencies_parsed_at":"2024-06-08T10:31:03.610Z","dependency_job_id":"42ed163d-4adc-4bca-a267-5fe74d59623d","html_url":"https://github.com/ymd-h/cpprb","commit_stats":null,"previous_names":[],"tags_count":158,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ymd-h%2Fcpprb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ymd-h%2Fcpprb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ymd-h%2Fcpprb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ymd-h%2Fcpprb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ymd-h","download_url":"https://codeload.github.com/ymd-h/cpprb/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247174415,"owners_count":20896078,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cython","machine-learning","python3","reinforcement-learning"],"created_at":"2024-11-14T12:39:01.427Z","updated_at":"2025-04-04T12:06:19.491Z","avatar_url":"https://github.com/ymd-h.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"#+OPTIONS: ':nil *:t -:t ::t \u003c:t H:3 \\n:nil ^:t arch:headline\n#+OPTIONS: author:t broken-links:nil c:nil creator:nil\n#+OPTIONS: d:(not \"LOGBOOK\") date:t e:t email:nil f:t inline:t num:t\n#+OPTIONS: p:nil pri:nil prop:nil stat:t tags:t tasks:t tex:t\n#+OPTIONS: timestamp:t title:t toc:nil todo:nil |:t\n#+TITLE: cpprb\n#+DATE: \u003c2019-01-29 Tue\u003e\n#+AUTHOR: Hiroyuki Yamada\n#+LANGUAGE: en\n#+SELECT_TAGS: export\n#+EXCLUDE_TAGS: noexport\n#+CREATOR: Emacs 26.1 (Org mode 9.1.14)\n\n#+HUGO_FRONT_MATTER_FORMAT: toml\n#+HUGO_CUSTOM_FRONT_MATTER: :disableToc true\n#+HUGO_LEVEL_OFFSET: 1\n#+HUGO_PRESERVE_FILLING:\n#+HUGO_DELETE_TRAILING_WS:\n#+HUGO_SECTION: .\n#+HUGO_BASE_DIR: ./site\n#+HUGO_DATE_FORMAT: %Y-%m-%dT%T+09:00\n#+HUGO_WEIGHT: auto\n\n#+STARTUP: showall logdone\n\n[[https://img.shields.io/gitlab/pipeline/ymd_h/cpprb.svg]]\n[[https://img.shields.io/pypi/v/cpprb.svg]]\n[[https://img.shields.io/pypi/l/cpprb.svg]]\n[[https://img.shields.io/pypi/status/cpprb.svg]]\n[[https://ymd_h.gitlab.io/cpprb/coverage/][https://gitlab.com/ymd_h/cpprb/badges/master/coverage.svg]]\n\n[[https://pypi.org/project/cpprb/][https://img.shields.io/pypi/dd/cpprb.svg]]\n[[https://pypi.org/project/cpprb/][https://img.shields.io/pypi/dw/cpprb.svg]]\n[[https://pypi.org/project/cpprb/][https://img.shields.io/pypi/dm/cpprb.svg]]\n\n\n#+attr_html: :width 100px\n[[https://ymd_h.gitlab.io/cpprb/images/favicon.png]]\n\n* Overview\n\ncpprb is a python ([[https://github.com/python/cpython/tree/master/Python][CPython]]) module providing replay buffer classes for\nreinforcement learning.\n\nMajor target users are researchers and library developers.\n\nYou can build your own reinforcement learning algorithms together with\nyour favorite deep learning library (e.g. [[https://www.tensorflow.org/][TensorFlow]], [[https://pytorch.org/][PyTorch]]).\n\ncpprb forcuses speed, flexibility, and memory efficiency.\n\nBy utilizing [[https://cython.org/][Cython]], complicated calculations (e.g. segment tree for\nprioritized experience replay) are offloaded onto C++.\n(The name cpprb comes from \"C++ Replay Buffer\".)\n\nIn terms of API, initially cpprb referred to [[https://github.com/openai/baselines][OpenAI Baselines]]'\nimplementation. The current version of cpprb has much more\nflexibility. Any [[https://numpy.org/][NumPy]] compatible types of any numbers of values can\nbe stored (as long as memory capacity is sufficient). For example, you\ncan store the next action and the next next observation, too.\n\n\n* DONE Installation\nCLOSED: [2020-01-17 Fri 23:08]\n:PROPERTIES:\n:EXPORT_HUGO_SECTION: installation\n:EXPORT_FILE_NAME: _index\n:EXPORT_HUGO_WEIGHT: 100\n:END:\n\ncpprb requires following softwares before installation.\n\n- C++17 compiler (for installation from source)\n  - [[https://gcc.gnu.org/][GCC]] (maybe 7.2 and newer)\n  - [[https://visualstudio.microsoft.com/][Visual Studio]] (2017 Enterprise is fine)\n- Python 3\n- pip\n\nAdditionally, here are user's good feedbacks for installation at [[https://gitlab.com/ymd_h/cpprb/issues/73][Ubuntu]].\n(Thanks!)\n\n\n** Install from [[https://pypi.org/][PyPI]] (Recommended)\n\nThe following command installs cpprb together with other dependencies.\n\n#+BEGIN_SRC shell\npip install cpprb\n#+END_SRC\n\nDepending on your environment, you might need =sudo= or =--user= flag\nfor installation.\n\nOn supported platflorms (Linux x86-64, Windows amd64, and macOS\nx86_64), binary packages hosted on PyPI can be used, so that you don't\nneed C++ compiler. On the other platforms, such as 32bit or\narm-architectured Linux and Windows, you cannot install from binary,\nand you need to compile by yourself. Please be patient, we plan to\nsupport wider platforms in future.\n\n\nIf you have any troubles to install from binary, you can fall back to\nsource installation by passing ~--no-binary~ option to the above pip\ncommand. (In order to avoid NumPy source installation, it is better to\ninstall NumPy beforehand.)\n\n#+begin_src shell\npip install numpy\npip install --no-binary cpprb\n#+end_src\n\n** Install from source code\n\nFirst, download source code manually or clone the repository;\n\n#+begin_src shell\ngit clone https://gitlab.com/ymd_h/cpprb.git\n#+end_src\n\nThen you can install in the same way;\n\n#+begin_src shell\ncd cpprb\npip install .\n#+end_src\n\n\nFor this installation, you need to convert extended Python (.pyx) to\nC++ (.cpp) during installation, it takes longer time than installation\nfrom PyPI.\n\n\n* DONE Usage\nCLOSED: [2020-01-17 Fri 23:09]\n:PROPERTIES:\n:EXPORT_HUGO_SECTION: features\n:EXPORT_FILE_NAME: simple_usage\n:END:\n\n** Basic Usage\nBasic usage is following step;\n1. Create replay buffer (~ReplayBuffer.__init__~)\n2. Add transitions (~ReplayBuffer.add~)\n   1. Reset at episode end (~ReplayBuffer.on_episode_end~)\n3. Sample transitions (~ReplayBuffer.sample~)\n\n\n** Example Code\nHere is a simple example for storing standard environment (aka. ~obs~,\n~act~, ~rew~, ~next_obs~, and ~done~).\n\n#+BEGIN_SRC python\nfrom cpprb import ReplayBuffer\n\nbuffer_size = 256\nobs_shape = 3\nact_dim = 1\nrb = ReplayBuffer(buffer_size,\n                  env_dict ={\"obs\": {\"shape\": obs_shape},\n                             \"act\": {\"shape\": act_dim},\n                             \"rew\": {},\n                             \"next_obs\": {\"shape\": obs_shape},\n                             \"done\": {}})\n\nobs = np.ones(shape=(obs_shape))\nact = np.ones(shape=(act_dim))\nrew = 0\nnext_obs = np.ones(shape=(obs_shape))\ndone = 0\n\nfor i in range(500):\n    rb.add(obs=obs,act=act,rew=rew,next_obs=next_obs,done=done)\n\n    if done:\n        # Together with resetting environment, call ReplayBuffer.on_episode_end()\n        rb.on_episode_end()\n\nbatch_size = 32\nsample = rb.sample(batch_size)\n# sample is a dictionary whose keys are 'obs', 'act', 'rew', 'next_obs', and 'done'\n#+END_SRC\n\n** Construction Parameters\n(See also [[https://ymd_h.gitlab.io/cpprb/api/api/cpprb.ReplayBuffer.html][API reference]])\n\n| Name             | Type                         | Optional           | Discription                       |\n|------------------+------------------------------+--------------------+-----------------------------------|\n| ~size~           | ~int~                        | No                 | Buffer size                       |\n| ~env_dict~       | ~dict~                       | Yes (but unusable) | Environment definition (See [[https://ymd_h.gitlab.io/cpprb/features/flexible_environment/][here]]) |\n| ~next_of~        | ~str~ or array-like of ~str~ | Yes                | Memory compression (See [[https://ymd_h.gitlab.io/cpprb/features/memory_compression/][here]])     |\n| ~stack_compress~ | ~str~ or array-like of ~str~ | Yes                | Memory compression (See [[https://ymd_h.gitlab.io/cpprb/features/memory_compression/][here]])     |\n| ~default_dtype~  | ~numpy.dtype~                | Yes                | Fall back data type               |\n| ~Nstep~          | ~dict~                       | Yes                | Nstep configuration (See [[https://ymd_h.gitlab.io/cpprb/features/nstep/][here]])    |\n| ~mmap_prefix~    | ~str~                        | Yes                | mmap file prefix (See [[https://ymd_h.gitlab.io/cpprb/features/mmap/][here]])       |\n\n\n** Notes\nFlexible environment values are defined by =env_dict= when buffer\ncreation. The detail is described at [[https://ymd_h.gitlab.io/cpprb/features/flexible_environment/][document]].\n\nSince stored values have flexible name, you have to pass to\n=ReplayBuffer.add= member by keyword.\n\n\n* DONE Features\nCLOSED: [2020-01-17 Fri 23:09]\n:PROPERTIES:\n:EXPORT_HUGO_SECTION: features\n:EXPORT_FILE_NAME: _index\n:EXPORT_HUGO_WEIGHT: 200\n:END:\n\ncpprb provides buffer classes for building following algorithms.\n\n| Algorithms                            | cpprb class                                  | Paper                  |\n|---------------------------------------+----------------------------------------------+------------------------|\n| Experience Replay                     | =ReplayBuffer=                               | [[https://link.springer.com/article/10.1007/BF00992699][L. J. Lin]]              |\n| [[https://ymd_h.gitlab.io/cpprb/features/per/][Prioritized Experience Replay]]         | =PrioritizedReplayBuffer=                    | [[https://arxiv.org/abs/1511.05952][T. Schaul et. al.]]      |\n| [[https://ymd_h.gitlab.io/cpprb/features/nstep/][Multi-step (Nstep) Learning]]           | =ReplayBuffer=, =PrioritizedReplayBuffer=    |                        |\n| [[https://ymd_h.gitlab.io/cpprb/features/ape-x/][Multiprocess Learning (Ape-X)]]         | =MPReplayBuffer= =MPPrioritizedReplayBuffer= | [[https://arxiv.org/abs/1803.00933][D. Horgan et. al.]]      |\n| [[https://ymd_h.gitlab.io/cpprb/features/laber/][Large Batch Experience Replay (LaBER)]] | =LaBERmean=, =LaBERlazy=, =LaBERmax=         | [[https://dblp.org/db/journals/corr/corr2110.html#journals/corr/abs-2110-01528][T. Lahire et al.]]       |\n| [[https://ymd_h.gitlab.io/cpprb/features/rer/][Reverse Experience Replay (RER)]]       | =ReverseReplayBuffer=                        | [[https://arxiv.org/abs/1910.08780][E. Rotinov]]             |\n| [[https://ymd_h.gitlab.io/cpprb/features/her/][Hindsight Experience Replay (HER)]]     | =HindsightReplayBuffer=                      | [[https://arxiv.org/abs/1707.01495][M. Andrychowicz et al.]] |\n\n\n\n\ncpprb features and its usage are described at following pages:\n- [[https://ymd_h.gitlab.io/cpprb/features/flexible_environment/][Flexible Environment]]\n- [[https://ymd_h.gitlab.io/cpprb/features/multistep_add/][Multi-step add]]\n- [[https://ymd_h.gitlab.io/cpprb/features/per/][Prioritized Experience Replay]]\n- [[https://ymd_h.gitlab.io/cpprb/features/nstep/][Nstep Experience Replay]]\n- [[https://ymd_h.gitlab.io/cpprb/features/memory_compression/][Memory Compression]]\n- [[https://ymd_h.gitlab.io/cpprb/features/mmap/][Map Large Data on File]]\n- [[https://ymd_h.gitlab.io/cpprb/features/ape-x/][Multiprocess Learning (Ape-X)]]\n- [[https://ymd_h.gitlab.io/cpprb/features/save_load_transitions/][Save/Load Transitions]]\n\n* DONE Design\nCLOSED: [2021-09-16 Thu 19:09]\n:PROPERTIES:\n:EXPORT_FILE_NAME: _index\n:EXPORT_HUGO_SECTION: design\n:EXPORT_HUGO_WEIGHT: 250\n:END:\n\n** Column-oriented and Flexible\nOne of the most distinctive design of cpprb is column-oriented\nflexibly defined transitions. As far as we know, other replay buffer\nimplementations adopt row-oriented flexible transitions (aka. array of\ntransition class) or column-oriented non-flexible transitions.\n\nIn deep reinforcement learning, sampled batch is divided into\nvariables (i.e. =obs=, =act=, etc.). If the sampled batch is\nrow-oriented, users (or library) need to convert it into\ncolumn-oriented one. (See [[https://ymd_h.gitlab.io/cpprb/features/flexible_environment/][doc]], too)\n\n** Batch Insertion\ncpprb can accept addition of multiple transitions simultaneously. This\ndesign is convenient when batch transitions are moved from local\nbuffers to a global buffer. Moreover it is more efficient because of\nnot only removing pure-Python =for= loop but also suppressing\nunnecessary priority updates for PER. (See [[https://ymd_h.gitlab.io/cpprb/features/multistep_add/][doc]], too)\n\n** Minimum Dependency\nWe try to minimize dependency. Only NumPy is required during its\nexecution. Small dependency is always preferable to avoid dependency\nhell.\n\n\n* DONE Contributing to cpprb\nCLOSED: [2020-01-17 Fri 23:09]\n:PROPERTIES:\n:EXPORT_HUGO_SECTION: contributing\n:EXPORT_FILE_NAME: _index\n:EXPORT_HUGO_WEIGHT: 300\n:END:\n\nAny contribution are very welcome!\n\n** Making Community Larger\nBigger commumity makes development more active and improve cpprb.\n\n- Star [[https://gitlab.com/ymd_h/cpprb][GitLab repository]] (and/or [[https://github.com/ymd-h/cpprb][GitHub Mirror]])\n- Publish your code using cpprb\n- Share this repository to your friend and/or followers.\n\n\n** Q \u0026 A at Forum\nWhen you have any problems or requests, you can check [[https://github.com/ymd-h/cpprb/discussions][Discussions on\nGitHub.com]]. If you still cannot find any information, you can post\nyour own.\n\nWe keep [[https://gitlab.com/ymd_h/cpprb/issues][issues on GitLab.com]] and users are still allowed to open\nissues, however, we mainly use the place as development issue tracker.\n\n** Merge Request (Pull Request)\n\ncpprb follows local rules:\n- Branch Name\n  - \"HotFix_***\" for bug fix\n  - \"Feature_***\" for new feature implementation\n- docstring\n  - Must for external API\n  - [[https://numpydoc.readthedocs.io/en/latest/format.html][Numpy Style]]\n- Unit Test\n  - Put test code under \"test/\" directory\n  - Can test by =python -m unittest \u003cYour Test Code\u003e= command\n  - Continuous Integration on GitLab CI configured by ~.gitlab-ci.yaml~\n- Open an issue and associate it to Merge Request\n\nStep by step instruction for beginners is described at [[https://ymd_h.gitlab.io/cpprb/contributing/merge_request][here]].\n\n* DONE Links\nCLOSED: [2020-01-17 Fri 23:09]\n:PROPERTIES:\n:EXPORT_HUGO_SECTION: misc\n:EXPORT_FILE_NAME: links\n:END:\n\n** cpprb sites\n- [[https://ymd_h.gitlab.io/cpprb/][Project Site]]\n  - [[https://ymd_h.gitlab.io/cpprb/api/][Class Reference]]\n  - [[https://ymd_h.gitlab.io/cpprb/coverage/][Unit Test Coverage]]\n- [[https://gitlab.com/ymd_h/cpprb][Main Repository]]\n- [[https://github.com/ymd-h/cpprb][GitHub Mirror]]\n- [[https://pypi.org/project/cpprb/][cpprb on PyPI]]\n\n** cpprb users' repositories\n- [[https://github.com/keiohta/tf2rl][keiohta/TF2RL]] :: TensorFlow2.x Reinforcement Learning\n\n** Example usage at Kaggle competition\n- [[https://www.kaggle.com/ymdhryk/ape-x-dqn-lap-safeguard-rewardredesign][Ape-X DQN-LAP: SafeGuard \u0026 RewardRedesign]] | [[https://www.kaggle.com/c/hungry-geese][Hungry Geese]]\n\n\n** Japanese Documents\n- [[https://qiita.com/ymd_h/items/505c607c40cf3e42d080][【強化学習】cpprb で Experience Replay を簡単に！| Qiita]]\n- [[https://qiita.com/ymd_h/items/ac9e3f1315d56a1b2718][【強化学習】Ape-X の高速な実装を簡単に！| Qiita]]\n- [[https://qiita.com/ymd_h/items/21071d7778cfb3cd596a][【強化学習】自作ライブラリでDQN | Qiita]]\n- [[https://zenn.dev/ymd_h/articles/03edcaa47a3b1c][【強化学習】Ape-Xの高速化を実現 | Zenn]]\n- [[https://zenn.dev/ymd_h/articles/e65fed3b7991c9][【強化学習】cpprb に遷移のファイル保存機能を追加 | Zenn]]\n\n* DONE License\nCLOSED: [2020-01-17 Fri 23:09]\n:PROPERTIES:\n:EXPORT_HUGO_SECTION: misc\n:EXPORT_FILE_NAME: license\n:END:\ncpprb is available under MIT license.\n\n#+INCLUDE: \"./LICENSE\" src plaintext\n\n\n* DONE Citation\nCLOSED: [2021-09-09 Thu 18:05]\n:PROPERTIES:\n:EXPORT_HUGO_SECTION: misc\n:EXPORT_FILE_NAME: citation\n:END:\n\nWe would be very happy if you cite cpprb in your papers.\n\n#+begin_src bibtex\n@misc{Yamada_cpprb_2019,\nauthor = {Yamada, Hiroyuki},\nmonth = {1},\ntitle = {{cpprb}},\nurl = {https://gitlab.com/ymd_h/cpprb},\nyear = {2019}\n}\n#+end_src\n\n\n- 3rd Party Papers citing cpprb\n  - [[https://www.sciencedirect.com/science/article/pii/S187705092100990X][E. Aitygulov and A. I. Panov, \"Transfer Learning with Demonstration Forgetting for Robotic Manipulator\", Proc. Comp. Sci. 186 (2021), 374-380, https://doi.org/10.1016/j.procs.2021.04.159]]\n  - [[https://nips.cc/Conferences/2021/Schedule?showEvent=21848][T. Kitamura and R. Yonetani, \"ShinRL: A Library for Evaluating RL Algorithms from Theoretical and Practical Perspectives\", NeurIPS Deep RL Workshop (2021)]] ([[https://arxiv.org/abs/2112.04123][arXiv]], [[https://github.com/omron-sinicx/ShinRL][code]])\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fymd-h%2Fcpprb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fymd-h%2Fcpprb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fymd-h%2Fcpprb/lists"}