{"id":15899755,"url":"https://github.com/yeyupiaoling/pytorch-ppo","last_synced_at":"2025-08-14T10:33:53.117Z","repository":{"id":110643129,"uuid":"315592979","full_name":"yeyupiaoling/Pytorch-PPO","owner":"yeyupiaoling","description":"基于Pytorch实现的PPO强化学习模型，支持训练各种游戏，如超级马里奥，雪人兄弟，魂斗罗等等。","archived":false,"fork":false,"pushed_at":"2021-02-19T12:22:24.000Z","size":80,"stargazers_count":22,"open_issues_count":2,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-02T16:53:59.163Z","etag":null,"topics":["ppo","pytroch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yeyupiaoling.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-11-24T10:13:21.000Z","updated_at":"2025-03-22T07:29:21.000Z","dependencies_parsed_at":"2023-07-08T13:16:15.569Z","dependency_job_id":null,"html_url":"https://github.com/yeyupiaoling/Pytorch-PPO","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/yeyupiaoling/Pytorch-PPO","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yeyupiaoling%2FPytorch-PPO","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yeyupiaoling%2FPytorch-PPO/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yeyupiaoling%2FPytorch-PPO/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yeyupiaoling%2FPytorch-PPO/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yeyupiaoling","download_url":"https://codeload.github.com/yeyupiaoling/Pytorch-PPO/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yeyupiaoling%2FPytorch-PPO/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270405936,"owners_count":24578154,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-14T02:00:10.309Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ppo","pytroch"],"created_at":"2024-10-06T10:40:24.642Z","updated_at":"2025-08-14T10:33:52.785Z","avatar_url":"https://github.com/yeyupiaoling.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Pytorch-PPO\n本项目是基于[Super-mario-bros-PPO-pytorch](https://github.com/uvipen/Super-mario-bros-PPO-pytorch) 复现的PPO，使用PPO模型训练Gym Retro游戏，支持训练各种游戏，如超级马里奥，雪人兄弟，魂斗罗等等。\n\n# 项目结构\n```shell script\nPPO-retro/\n├── actions.py        定义每个游戏的动作\n├── discretizer.py    定义有点动作的工具类\n├── env.py            用于游戏环境和多线程游戏环境\n├── infer.py          使用训练好的模型进行推理\n├── model.py          模型结构\n├── retro_util.py     retro游戏动作和图像处理\n├── retrowrapper.py   用于支持多线程retro游戏环境\n├── test_env.py       测试游戏环境\n├── train.py          训练PPO模型\n└── utils.py          用于评估和输出参数\n```\n\n# retro游戏模拟器\nGym Retro可的游戏转变为Gym环境以进行强化学习，并附带约1000种游戏的集成功能。它使用各种支持Libretro API的仿真器，从而使添加新仿真器变得相当容易。\n支持平台：\n- Windows 7、8、10\n- macOS 10.13（High Sierra），10.14（Mojave）\n- Linux（许多Linux1）\n\n支持的Python：\n- 3.6\n- 3.7\n- 3.8\n\n每个游戏集成都具有列出游戏中变量的存储位置的文件，基于这些变量的奖励功能，情节结束条件。\n\n**请注意** ，不包含ROM，必须自己获取它们。\n\n1. 安装retro\n```shell\npip install gym-retro -i https://mirrors.aliyun.com/pypi/simple/\n```\n\n2. 查看retro支持的游戏，执行这两行代码会输出retro支持的游戏。\n```python\nimport retro\nretro.data.list_games()\n```\n\n3. 导入游戏的ROM文件，我们安装了retro不是就能够直接使用的，因为retro不会提供游戏的ROM文件，所以需要我们自己去下载和导入。跟着笔者下载游戏的ROM文件并导入到Python环境中。[这里](https://archive.org/download/No-Intro-Collection_2016-01-03_Fixed) 提供了很多游戏的ROM，笔者下载的是[Nintendo - Nintendo Entertainment System.zip](https://archive.org/download/No-Intro-Collection_2016-01-03_Fixed/Nintendo%20-%20Nintendo%20Entertainment%20System.zip) ，导入命令如下：\n```shell script\npython -m retro.import /ROM的文件夹/\n```\n**特别注意：** 如果ROM带有`.bin`扩展名，需要将其重命名为具有该系统正确的扩展名。如Atari类型的，需要改成`.a26`，以便`python -m retro.import`成功！\n```shell script\n.md： 世嘉创世纪（Mega Drive）\n.sfc：超级任天堂娱乐系统\n.nes：任天堂娱乐系统\n.a26：Atari 2600\n.gb： 任天堂游戏男孩\n.gba：任天堂Game Boy Advance\n.gbc：任天堂游戏男孩Color\n.gg： 世嘉游戏装备\n.pce：NEC TurboGrafx-16\n.sms：世嘉Master System\n```\n\n4. 测试游戏环境，通过执行`test_env.py`可以测试游戏的环境。\n```python\nimport cv2\nimport numpy as np\n\nfrom src.env import create_train_env\n\n\ndef main():\n    # 获取游戏\n    env = create_train_env(game=\"SuperMarioBros-Nes\")\n    print(env.observation_space.shape)\n    print(env.action_space.n)\n\n    obs = env.reset()\n\n    while True:\n        # 游戏生成的随机动作，int类型数值\n        action = env.action_space.sample()\n        # 执行游戏\n        obs, reward, terminal, info = env.step(action)\n        # 显示连续动作\n        obs = np.squeeze(obs)\n        obses = obs[0]\n        for i in range(1, obs.shape[0]):\n            obses = np.hstack([obses, obs[i]])\n        cv2.imshow('obes', obses)\n        cv2.waitKey(1)\n        env.render()\n        print(\"=\" * 50)\n        print(\"action:\", action)\n        print(\"obs shape:\", obs.shape)\n        print(\"reward:\", reward)\n        print(\"terminal:\", terminal)\n        print(\"info:\", info)\n        if terminal:\n            obs = env.reset()\n\n\nif __name__ == \"__main__\":\n    main()\n```\n\n# 训练\n\n1. 首先安装PaddlePaddle以及相关库。\n```shell\npip install torch==1.5.1+cu101 torchvision==0.6.1+cu101 -f https://download.pytorch.org/whl/torch_stable.html\n```\n\n2. 执行训练，通过game参数可以指定想要选择的游戏，前提是retro得支持这个游戏并且已经导入了ROM文件。\n```shell\npython train.py --game=SuperMarioBros-Nes\n```\n\n3. 验证通过的游戏如下，最新的验证通过的游戏请参考本项目的Github文档。\n```shell script\nSuperMarioBros-Nes\nSnowBrothers-Nes\n```\n\n4. 自行训练其他游戏，理论情况下只需要更改游戏名称就可以正常训练，但是为了模型更好的拟合，最好是可以修改一下游戏可执行的动作，动作在`actions.py`。\n\n## 预测\n预测程序会使用训练时保存的最好得分模型进行预测，这个预测程序需要在界面环境下执行，如果要在终端下执行，需要注释`env.render()`这行代码。\n```shell\npython infer.py --game=SuperMarioBros-Nes\n```\n\n# 模型下载\n\n| 游戏名称 | 预训练模型 | 最好的通关模型 |\n| :---: | :---: | :---: |\n| SuperMarioBros-Nes（超级马里奥） | [点击下载](https://resource.doiduoyi.com/#y384501) | [点击下载](https://resource.doiduoyi.com/#y384501) |\n| SnowBrothers-Nes（雪人兄弟） | [点击下载](https://resource.doiduoyi.com/#y384501) | [点击下载](https://resource.doiduoyi.com/#y384501) |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyeyupiaoling%2Fpytorch-ppo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyeyupiaoling%2Fpytorch-ppo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyeyupiaoling%2Fpytorch-ppo/lists"}