{"id":13958513,"url":"https://github.com/huawei-noah/xingtian","last_synced_at":"2025-04-05T23:11:36.524Z","repository":{"id":42074532,"uuid":"287759621","full_name":"huawei-noah/xingtian","owner":"huawei-noah","description":"xingtian is a componentized library for the development and verification of reinforcement learning algorithms","archived":false,"fork":false,"pushed_at":"2023-09-12T12:56:36.000Z","size":7392,"stargazers_count":311,"open_issues_count":12,"forks_count":89,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-03-29T22:08:54.208Z","etag":null,"topics":["dqn","impala","muzero","ppo","qmix","reinforcement-learning-algorithms"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/huawei-noah.png","metadata":{"files":{"readme":"README.cn.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-08-15T14:13:06.000Z","updated_at":"2025-03-23T03:35:43.000Z","dependencies_parsed_at":"2024-12-16T14:43:15.756Z","dependency_job_id":"9ffc8b1a-cff0-48fe-888c-81ec2e97b870","html_url":"https://github.com/huawei-noah/xingtian","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huawei-noah%2Fxingtian","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huawei-noah%2Fxingtian/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huawei-noah%2Fxingtian/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huawei-noah%2Fxingtian/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/huawei-noah","download_url":"https://codeload.github.com/huawei-noah/xingtian/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247411239,"owners_count":20934653,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dqn","impala","muzero","ppo","qmix","reinforcement-learning-algorithms"],"created_at":"2024-08-08T13:01:41.263Z","updated_at":"2025-04-05T23:11:36.504Z","avatar_url":"https://github.com/huawei-noah.png","language":"Python","funding_links":[],"categories":["时间序列"],"sub_categories":["网络服务_其他"],"readme":"\n\u003cdiv align=\"center\"\u003e\n\u003cimg width=\"300px\" height=\"auto\" src=\"./docs/.images/xingtian-logo.png\"\u003e\n\u003c/div\u003e\n\n[English](./README.md)\n\n## 简介 \n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n**刑天 (XingTian)** 是一个组件化强化学习库，用于开发、验证强化学习算法。它目前已支持包括DQN、DDPG、PPO和IMPALA等系列算法，可以在多种环境中训练智能体，如Gym、Atari、Torcs、StarCraftII等。 为了满足用户快速验证和解决RL问题的需求，刑天抽象出了四个模块：`Algorithm`,`Model`,`Agent`,`Environment`。它们的工作方式类似于\"乐高\"积木的组合。更详细的内容请[阅读架构介绍](./docs/basic_arch.cn.md). \n\n## 系统依赖\n\n```shell\n# ubuntu 18.04\nsudo apt-get install python3-pip libopencv-dev -y\npip3 install opencv-python\n\n# Run with tensorflow 1.15.0 or tensorflow 2.3.1\npip3 install zmq h5py gym[atari] tqdm imageio matplotlib==3.0.3 Ipython pyyaml tensorflow==1.15.0 pyarrow lz4 fabric2 absl-py psutil tensorboardX setproctitle\n```\n\n也可使用pip 进行依赖安装 `pip3 install -r requirements.txt`\n\n如果需要使用Pytorch 作为后端引擎，请自行安装.  [Ref Pytorch](https://pytorch.org/get-started/locally/)\n\n\n\n\n## 安装\n```zsh\n# cd PATH/TO/XingTian \npip3 install -e .\n```\n\n可通过 `import xt; print(xt.__Version__)`  来确认是否已正常安装. \n\n```python\nIn [1]: import xt\n\nIn [2]: xt.__version__\nOut[2]: '0.3.0'\n```\n\n\n\n## 快速开始\n\n---------\n#### 参数配置\n下面是一个有关 [倒立摆](https://gym.openai.com/envs/CartPole-v0/) 简单任务的参数示例，我们通过配置系统中已注册的算法，环境信息来组合训练任务。有关不同参数更详细的描述可以在[用户指导](./docs/user.cn.md) 中找到。\n\n\n```yaml\nalg_para:\n  alg_name: PPO\n  alg_config:\n    process_num: 1\n    save_model: True  # default False\n    save_interval: 100\n\nenv_para:\n  env_name: GymEnv\n  env_info:\n    name: CartPole-v0\n    vision: False\n\nagent_para:\n  agent_name: PPO\n  agent_num : 1\n  agent_config:\n    max_steps: 200\n    complete_step: 1000000\n    complete_episode: 3550\n\nmodel_para:\n  actor:\n    model_name: PpoMlp\n    state_dim: [4]\n    action_dim: 2\n    input_dtype: float32\n    model_config:\n      BATCH_SIZE: 200\n      CRITIC_LOSS_COEF: 1.0\n      ENTROPY_LOSS: 0.01\n      LR: 0.0003\n      LOSS_CLIPPING: 0.2\n      MAX_GRAD_NORM: 5.0\n      NUM_SGD_ITER: 8\n      SUMMARY: False\n      VF_SHARE_LAYERS: False\n      activation: tanh\n      hidden_sizes: [64, 64]\n\nenv_num: 10\n```\n\n另外在 [examples](./examples) 目录下，可以找到更加丰富的训练配置示例。\n\n#### 开始训练任务\n\n```python3 xt/main.py -f examples/cartpole_ppo.yaml -t train```\n\n![img](./docs/.images/cartpole.gif)  \n\n\n\n#### 评估本机模型\n\n在你的`.yaml`文件中设置 `benchmark.eval.model_path`  参数，然后通过 `-t evaluate` 运行评估任务。\n\n```\nbenchmark:\n  eval:\n    model_path: /YOUR/PATH/TO/EVAL/models\n    gap: 10           # 目录下需评估模型的间隔\n    evaluator_num: 1  # 启动评估实例的数量，可支持并行评估\n\n# 运行命令\npython3 xt/main.py -f examples/cartpole_ppo.yaml -t evaluate\n```\n\n\u003e 系统默认启动训练任务，即 -t 的默认选项是 train\n\n#### 使用命令行\n\n```zsh\n# 在终端中，可直接使用xt_main 替换 python3 xt/main.py 执行命令\nxt_main -f examples/cartpole_ppo.yaml -t train\n\n# train with evaluate\nxt_main -f examples/cartpole_ppo.yaml -t train_with_evaluate\n```\n\n## 自定义任务的开发\n\n1. 编写自定义模块，并注册。 具体可参考 [开发指导](./docs/developer.cn.md)\n2. 在配置文件 `your_train_configure.yaml`中，配置自定义的模块名字\n3.  启动训练  `xt_main -f path/to/your_train_configure.yaml` :)\n\n## 实验结果参考\n\n#### 平均的训练回报\n\n1. 10M step 之后的**DQN** 收敛回报 (**40M frames**).\n\n   | env           | XingTian Basic DQN | RLlib Basic DQN | Hessel et al. DQN |\n   | ------------- | ------------------ | --------------- | ----------------- |\n   | BeamRider     | 6706               | 2869            | ~2000             |\n   | Breakout      | 352                | 287             | ~150              |\n   | QBert         | 14087              | 3921            | ~4000             |\n   | SpaceInvaders | 947                | 650             | ~500              |\n\n2. 10M step 之后的**PPO** 收敛回报 (**40M frames**).\n\n   | env           | XingTian PPO | RLlib PPO | Baselines PPO |\n   | ------------- | ------------ | --------- | ------------- |\n   | BeamRider     | 4877         | 2807      | ~1800         |\n   | Breakout      | 341          | 104       | ~250          |\n   | QBert         | 14771        | 11085     | ~14000        |\n   | SpaceInvaders | 1025         | 671       | ~800          |\n\n3. 10M step 之后的**IMPALA** 收敛回报 (**40M frames**).\n\n   | env           | XingTian IMPALA | RLlib IMPALA |\n   | ------------- | --------------- | ------------ |\n   | BeamRider     | 2313            | 2071         |\n   | Breakout      | 334             | 385          |\n   | QBert         | 12205           | 4068         |\n   | SpaceInvaders | 742             | 719          |\n\n\n\n#### 吞吐量\n\n1. **DQN**\n\n   | env           | XingTian Basic DQN | RLlib Basic DQN |\n   | ------------- | ------------------ | --------------- |\n   | BeamRider     | 129                | 109             |\n   | Breakout      | 117                | 113             |\n   | QBert         | 111                | 90              |\n   | SpaceInvaders | 115                | 100             |\n\n\n2. **PPO**\n\n   | env           | XingTian PPO | RLlib PPO |\n   | ------------- | ------------ | --------- |\n   | BeamRider     | 2422         | 1618      |\n   | Breakout      | 2497         | 1535      |\n   | QBert         | 2436         | 1617      |\n   | SpaceInvaders | 2438         | 1608      |\n\n3. **IMPALA**\n\n   | env           | XingTian IMPALA | RLlib IMPALA |\n   | ------------- | --------------- | ------------ |\n   | BeamRider     | 8756            | 3637         |\n   | Breakout      | 8814            | 3525         |\n   | QBert         | 8249            | 3471         |\n   | SpaceInvaders | 8463            | 3555         |\n\n\u003e 实验硬件环境： 72  Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz with single Tesla V100\n\u003e\n\u003e Ray reward数据来自 [https://github.com/ray-project/rl-experiments](https://github.com/ray-project/rl-experiments), 吞吐量来自以上硬件设备的测试数据\n\n## 致谢\n\n刑天参考了以下项目: [DeepMind/scalable_agent](https://github.com/deepmind/scalable_agent), [baselines](https://github.com/openai/baselines), [ray](https://github.com/ray-project/ray).\n\n## 许可证\n\nThe MIT License(MIT)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuawei-noah%2Fxingtian","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhuawei-noah%2Fxingtian","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuawei-noah%2Fxingtian/lists"}