{"id":18551329,"url":"https://github.com/morvanzhou/rlearn","last_synced_at":"2025-04-09T22:31:33.740Z","repository":{"id":194694219,"uuid":"584627033","full_name":"MorvanZhou/rlearn","owner":"MorvanZhou","description":"A Python Reinforcement Learning Package","archived":false,"fork":false,"pushed_at":"2023-09-14T04:22:31.000Z","size":1206,"stargazers_count":8,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-24T12:55:32.575Z","etag":null,"topics":["reinforcement-learning","rl"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MorvanZhou.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-01-03T05:05:18.000Z","updated_at":"2025-02-03T13:01:22.000Z","dependencies_parsed_at":"2023-09-14T18:15:49.299Z","dependency_job_id":"ad28f7c1-6935-4eb7-8462-25292d374a78","html_url":"https://github.com/MorvanZhou/rlearn","commit_stats":null,"previous_names":["morvanzhou/rlearn"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MorvanZhou%2Frlearn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MorvanZhou%2Frlearn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MorvanZhou%2Frlearn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MorvanZhou%2Frlearn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MorvanZhou","download_url":"https://codeload.github.com/MorvanZhou/rlearn/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248123593,"owners_count":21051500,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["reinforcement-learning","rl"],"created_at":"2024-11-06T21:08:40.132Z","updated_at":"2025-04-09T22:31:32.894Z","avatar_url":"https://github.com/MorvanZhou.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Reinforcement learning Algorithms\n\n- On-Policy\n  - A2C: Actor-Critic\n  - PPO: Proximal Policy Optimization\n- Off-Policy\n  - DQN: Deep Q Networks\n  - DuelingDQN: Dueling DQN\n  - DDPG: Deep Deterministic Policy Gradients\n  - TD3: Twin Delayed DDPG\n  - SAC: Soft Actor Critic\n\n# Install\n\n```shell\npip install rlearn\n```\n\n```shell\ngit clone https://git.woa.com/TIPE/rlearn.git\ncd rlearn\n\n# apple m1 silicon should use conda command:\nconda install -c apple tensorflow-deps\n########\n\npython3 setup.py install\n```\n\n# Usage\n\n## Classical way\n\n```python\nimport gymnasium\nfrom tensorflow import keras\n\nimport rlearn\n\n# define an environment\nenv = gymnasium.make('CartPole-v1', render_mode=\"human\")\n\n# set reinforcement learning trainer\ntrainer = rlearn.DQNTrainer()\ntrainer.set_replay_buffer(max_size=1000)\ntrainer.set_model_encoder(\n  q=keras.Sequential([\n    keras.layers.InputLayer(4),  # state has dimension of 4\n    keras.layers.Dense(32),\n    keras.layers.ReLU(),\n  ]),\n  action_num=env.action_space.n\n)\n\n# training loop\nfor _ in range(100):\n  s, _ = env.reset()\n  for _ in range(200):\n    a = trainer.predict(s)\n    s_, r, done, _, _ = env.step(a)\n    trainer.store_transition(s, a, r, s_, done)\n    trainer.train_batch()\n    s = s_\n    if done:\n      break\n```\n\nset training hyper parameters\n\n```python\nimport rlearn\n\ntrainer = rlearn.DQNTrainer()\ntrainer.set_params(\n  learning_rate=0.01,\n  batch_size=32,\n  gamma=0.9,\n  replace_ratio=1.,\n  replace_step=0,\n  min_epsilon=0.1,\n  epsilon_decay=1e-3,\n)\n```\n\n## Parallel training\n\n### experience parallel\n\nStart a remote buffer:\n\n```python\nfrom rlearn import distributed\n\ndistributed.experience.start_replay_buffer_server(\n  port=50051,\n)\n```\n\nStart actors:\n\n```python\nfrom rlearn import distributed\nimport gymnasium\n\n\nclass CartPole(rlearn.EnvWrapper):\n  def __init__(self, render_mode=\"human\"):\n    self.env = gymnasium.make('CartPole-v1', render_mode=render_mode)\n\n  def reset(self):\n    s, _ = self.env.reset()\n    return s\n\n  def step(self, a):\n    s_, _, done, _, _ = self.env.step(a)\n    r = -1 if done else 0\n    return s_, r, done\n\n\ndistributed.experience.start_actor_server(\n  port=50052,\n  remote_buffer_address=\"localhost:50051\",\n  env=CartPole(),\n)\n```\n\nStart a learner:\n\n```python\nimport rlearn\nfrom tensorflow import keras\n\ntrainer = rlearn.trainer.DQNTrainer()\ntrainer.set_model_encoder(\n  q=keras.Sequential([\n    keras.layers.InputLayer(4),\n    keras.layers.Dense(32),\n    keras.layers.ReLU(),\n  ]),\n  action_num=2\n)\ntrainer.set_params(\n  learning_rate=0.01,\n  batch_size=32,\n  replace_step=15,\n)\ntrainer.set_action_transformer(rlearn.transformer.DiscreteAction([0, 1]))\nlearner = rlearn.distributed.experience.Learner(\n  trainer=trainer,\n  remote_buffer_address=\"localhost:50051\",\n  actors_address=[\"localhost:50052\", ],\n  actor_buffer_size=10,\n  remote_buffer_size=1000,\n  remote_buffer_type=\"RandomReplayBuffer\",\n)\nlearner.run(max_train_time=100, max_ep_step=-1)\n```\n\n### gradient parallel\n\nStart a parameter server\n\n```python\nimport rlearn\n\ntrainer = rlearn.trainer.DQNTrainer()\ntrainer.set_model_encoder(\n  q=keras.Sequential([\n    keras.layers.InputLayer(4),\n    keras.layers.Dense(20),\n    keras.layers.ReLU(),\n  ]),\n  action_num=2\n)\ntrainer.set_params(\n  learning_rate=0.001,\n  batch_size=32,\n  replace_step=100,\n)\ntrainer.set_action_transformer(rlearn.transformer.DiscreteAction([0, 1]))\n\nrlearn.distributed.gradient.start_param_server(\n  port=50051,\n  trainer=trainer,\n  sync_step=5,\n  worker_buffer_type=\"RandomReplayBuffer\",\n  worker_buffer_size=3000,\n  max_train_time=60,\n  # debug=True,\n)\n```\n\nStart workers\n\n```python\nimport gymnasium\nimport rlearn\n\n\nclass CartPole(rlearn.EnvWrapper):\n  def __init__(self, render_mode=\"human\"):\n    self.env = gymnasium.make('CartPole-v1', render_mode=render_mode)\n\n  def reset(self):\n    s, _ = self.env.reset()\n    return s\n\n  def step(self, a):\n    s_, _, done, _, _ = self.env.step(a)\n    r = -1 if done else 0\n    return s_, r, done\n\n\nrlearn.distributed.gradient.worker.run(\n  env=CartPole(),\n  params_server_address=\"localhost:50051\",\n  name=\"worker1\",\n  # debug=True,\n)\n```\n\n## Save and reload\n\nSave entire model\n\n```python\nimport rlearn\nfrom tensorflow import keras\nimport numpy as np\n\n# define and save a model\ntrainer = rlearn.DQNTrainer()\ntrainer.set_model_encoder(\n  keras.Sequential([\n    keras.layers.InputLayer(2),\n    keras.layers.Dense(32),\n  ]), action_num=3)\npath = \"tmp_model0\"\ntrainer.save_model(path)\n\n# reload directory from path\nm = rlearn.load_model(path)\naction = m.predict(np.random.random((2,)))\n```\n\nSave model parameters and reload to a new trainer or new model.\n\n```python\nimport rlearn\nfrom tensorflow import keras\nimport numpy as np\n\n# define and save a model\ntrainer = rlearn.DQNTrainer()\ntrainer.set_model_encoder(\n  keras.Sequential([\n    keras.layers.InputLayer(2),\n    keras.layers.Dense(32),\n  ]), action_num=3)\npath = \"tmp_model_weights0\"\ntrainer.save_model_weights(path)\n\n# trainer load parameters from path\ntrainer2 = rlearn.DQNTrainer()\ntrainer2.set_model_encoder(\n  keras.Sequential([\n    keras.layers.InputLayer(2),\n    keras.layers.Dense(32),\n  ]), action_num=3)\ntrainer2.load_model_weights(path)\naction = trainer2.predict(np.random.random((2,)))\n\n# model load parameters\nm = rlearn.DQN()\nm.set_encoder(encoder=keras.Sequential([\n  keras.layers.InputLayer(2),\n  keras.layers.Dense(32),\n]), action_num=3)\naction = m.predict(np.random.random((2,)))\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmorvanzhou%2Frlearn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmorvanzhou%2Frlearn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmorvanzhou%2Frlearn/lists"}