{"id":13745541,"url":"https://github.com/tencent-ailab/hok_env","last_synced_at":"2025-05-16T16:08:07.480Z","repository":{"id":39409965,"uuid":"501082698","full_name":"tencent-ailab/hok_env","owner":"tencent-ailab","description":"Honor of Kings AI Open Environment of Tencent","archived":false,"fork":false,"pushed_at":"2024-07-17T02:52:45.000Z","size":45109,"stargazers_count":719,"open_issues_count":16,"forks_count":83,"subscribers_count":17,"default_branch":"master","last_synced_at":"2025-05-14T21:26:00.918Z","etag":null,"topics":["reinforcement-learning"],"latest_commit_sha":null,"homepage":"https://aiarena.tencent.com/aiarena/en/open-gamecore","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tencent-ailab.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-06-08T03:13:34.000Z","updated_at":"2025-05-14T13:50:28.000Z","dependencies_parsed_at":"2024-01-10T21:05:16.952Z","dependency_job_id":null,"html_url":"https://github.com/tencent-ailab/hok_env","commit_stats":{"total_commits":46,"total_committers":4,"mean_commits":11.5,"dds":"0.23913043478260865","last_synced_commit":"26d455f6136ea8c6e5b73cf9299f7966ca2dccf5"},"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tencent-ailab%2Fhok_env","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tencent-ailab%2Fhok_env/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tencent-ailab%2Fhok_env/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tencent-ailab%2Fhok_env/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tencent-ailab","download_url":"https://codeload.github.com/tencent-ailab/hok_env/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254564127,"owners_count":22092122,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["reinforcement-learning"],"created_at":"2024-08-03T06:00:23.884Z","updated_at":"2025-05-16T16:08:07.453Z","avatar_url":"https://github.com/tencent-ailab.png","language":"Python","funding_links":[],"categories":["Reinforcement Learning \u003ca name=\"reinforcementlearning\"\u003e\u003c/a\u003e","时间序列"],"sub_categories":["Environments","网络服务_其他"],"readme":"# Honor of Kings AI Open Environment of Tencent(腾讯王者荣耀AI开放环境)\n\n![avatar](./docs/hok_1v1.png)\n\n## Update: 3v3 Mode Now Available\n\n```\npython3.8 -c \"from hok.hok3v3.unit_test.test_env import run_test; run_test()\"\n```\n\nPlease refer to [hok3v3](https://doc.aiarena.tencent.com/paper/hok3v3/latest/hok3v3_env/honor-of-kings/) for further information.\n\nPlease consult [cluster.md](docs/cluster.md) document for instructions on cluster training utilizing the `hok_env` environment and the integrated `rl_framework`.\n\n## Introduction\n\n[![PyPI](https://github.com/tencent-ailab/hok_env/actions/workflows/python-publish.yml/badge.svg)](https://github.com/tencent-ailab/hok_env/actions/workflows/python-publish.yml)\n[![Image](https://github.com/tencent-ailab/hok_env/actions/workflows/gpu.yml/badge.svg)](https://github.com/tencent-ailab/hok_env/actions/workflows/gpu.yml)\n\n- [Hok_env](https://github.com/tencent-ailab/hok_env) is the open environment of the MOBA game: [Honor of kings](https://pvp.qq.com/).\n\n- This repository mainly includes Hok_env SDK, a reinforcement learning training framework and an implementation of ppo algorithm based on the training framework. Hok_env SDK is used to interact with the gamecore of Honor of Kings.\n\n- This repository also contains the implementation code for the paper:\n\n    \u003e **Honor of Kings Arena: an Environment for Generalization in Competitive Reinforcement Learning**.\\\n    \u003e Hua Wei*, Jingxiao Chen*, Xiyang Ji*, Hongyang Qin, Minwen Deng, Siqin Li, Liang Wang, Weinan Zhang, Yong Yu, Lin Liu, Lanxiao Huang, Deheng Ye, Qiang Fu, Wei Yang. (*Equal contribution) \\\n    \u003e **NeurIPS Datasets and Benchmarks 2022** \\\n    \u003e Project Page: https://github.com/tencent-ailab/hok_env \\\n    \u003e arXiv: https://arxiv.org/abs/2209.08483\n\n    \u003e **Abstract**: *This paper introduces Honor of Kings Arena, a reinforcement learning (RL) environment based on Honor of Kings, one of the world’s most popular games at present. Compared to other environments studied in most previous work, ours presents new generalization challenges for competitive reinforcement learning. It is a multiagent problem with one agent competing against its opponent; and it requires the generalization ability as it has diverse targets to control and diverse opponents to compete with. We describe the observation, action, and reward specifications for the Honor of Kings domain and provide an open-source Python-based interface for communicating with the game engine. We provide twenty target heroes with a variety of tasks in Honor of Kings Arena and present initial baseline results for RL-based methods with feasible computing resources. Finally, we showcase the generalization challenges imposed by Honor of Kings Arena and possible remedies to the challenges. All of the software, including the environment-class, are publicly available at: https://github.com/tencent-ailab/hok_env. The documentation is available at: https://aiarena.tencent.com/hok/doc/.*\n\n- Current supported heroes in hok_env:\n  - lubanqihao\n  - miyue\n  - libai\n  - makeboluo (Marco Polo)\n  - direnjie\n  - guanyu\n  - diaochan\n  - luna\n  - hanxin\n  - huamulan\n  - buzhihuowu (Mai Shiranui)\n  - jvyoujing (Ukyou Tachibana)\n  - houyi\n  - zhongkui\n  - ganjiangmoye\n  - kai\n  - gongsunli\n  - peiqinhu\n  - shangguanwaner\n\n\n## Running Requirement\n\n- python \u003e= 3.6, \u003c= 3.9.\n\n- Windows 10/11 or Linux wine (to deploy windows gamecore server)\n\n- Docker (to deploy hok_env on Linux containers)\n\n  - For windows, WSL 2 is required. (Windows Subsystem for Linux Version 2.0)\n\nThe gamecore of hok_env runs on the Windows platform, and the package **_hok_env_** needs to be deployed in linux platforms to interact with the gamecore.\n\nWe also provided a docker image for training on your computer. In a further version, we will release a gamecore server compatible with linux.\n\nTo enable cluster training, here is a workaround by running Windows gamecore on Linux: [run windows gamecore on linux](./docs/run_windows_gamecore_on_linux.md).\n\n## Gamecore Installation\n\n### Download the hok gamecore\n\nYou need to apply for the license and gamecore on this page: https://aiarena.tencent.com/aiarena/en/open-gamecore\n\nPlease put the `license.dat` under the folder:`hok_env_gamecore/gamecore/core_assets`.\n\n![avatar](./docs/sgame_folder.png)\n\n### Test the gamecore\n\nopen CMD\n```shell\n cd gamecore\\bin\n set PATH=%PATH%;..\\lib\\\n .\\sgame_simulator_remote_zmq.exe .\\sgame_simulator.common.conf\n```\n\n`sgame_simulator_remote_zmq.exe` requires one parameters: `sgame_simulator.common.conf` the config file path\n\nYou can see the following message:\n```\nPlayerNum:2\nAbsPath:../scene/1V1.abs\nPlayerInfo [CampID:0][HeroID:199][Skill:80104][AutoAi:1][AiServer::0:100] [Symbol 0 0 0] [Request:-1]\nPlayerInfo [CampID:1][HeroID:199][Skill:80104][AutoAi:1][AiServer::0:100] [Symbol 0 0 0] [Request:-1]\nSGame Simulator Begin\ninit_ret:0\nseed: 3417\nSymbols:\ninHeroId: 199\n0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\ninHeroId: 199\n0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\nstart_ret:0\nHero[0] inHeroId:199; outPlayerId:148\nHero[1] inHeroId:199; outPlayerId:149\n[Hero Info] [HeroID:199] [RuntimeID:8] client_id:0.0.0.0_1658_0_20221202193534_148\n[Hero Info] [HeroID:199] [RuntimeID:9] client_id:0.0.0.0_1658_0_20221202193534_149\nboost_ret finished: 8, gameover_ai_server: 0\nclose_ret:0\nuninit_ret:0\nSGame Simulator End [FrameNum:8612][TimeUsed:7580ms]\n```\nThe gamecore has started successfully!\n\nHere is the content of `sgame_simulator.common.conf`:\n```json\n{\n    \"abs_file\": \"../scene/1V1.abs\",\n    \"core_assets\": \"../core_assets\",\n    \"game_id\": \"kaiwu-base-35401-35400-6842-1669963820108111766-217\",\n    \"hero_conf\": [\n        {\n            \"hero_id\": 199\n        },\n        {\n            \"hero_id\": 199\n        }\n    ]\n}\n```\n\noutput files:\n```\nAIOSS_221202-1935_linux_1450111_1450123_1_1_20001_kaiwu-base-35401-35400-6842-1669963820108111766-217.abs\nkaiwu-base-35401-35400-6842-1669963820108111766-217.json\nkaiwu-base-35401-35400-6842-1669963820108111766-217.stat\nkaiwu-base-35401-35400-6842-1669963820108111766-217_detail.stat\n```\n\n## 1v1\n\n### Observation and action spaces\n\nPlease refer to https://aiarena.tencent.com/hok/doc/quickstart/index.html\n\n### Usage\n\nPlease refer to [hok1v1/unit_test](hok_env/hok/hok1v1/unit_test/test_env.py) for the basic usage of the hok1v1.\n\nPlease refer to [aiarena/1v1](aiarena/1v1) for the training code of hok1v1.\n\n### Test the gamecore with the demo script in WSL\n\nYou can test gamecore with a simple python script in wsl.\n\n#### Make sure your pc supports wsl2\n\nFor the installation and upgrade of wsl2, please refer to the link: https://docs.microsoft.com/zh-cn/windows/wsl/install-manual#step-4---download-the-linux-kernel-update-package`\n\nYou need to install python3.6 and some required dependencies in wsl.\n\n#### Run the test script in wsl2\n\n0. Start the gamecore server outside wsl2\n\n    ```shell\n    cd gamecore\n    gamecore-server.exe server --server-address :23432\n    ```\n\n1. Install hok_env in python\n\n    ```shell\n    ## after git clone this repo \n    cd hok_env/hok_env\n    pip install -e .\n    ```\n\n2. Run the test script\n\n    ```shell\n    cd /hok_env/hok/hok1v1/unit_test\n    python test_env.py\n    ```\n\nIf you see the following message, you have successfully established a connection with Hok_env and have completed a game. Congratulations!\n\n```\n# python test_env.py\n127.0.0.1:23432 127.0.0.1\n======= test_send_action\ncamp_config {'mode': '1v1', 'heroes': [[{'hero_id': 132}], [{'hero_id': 133}]]}\ncommon_ai [False, True]\ntry to get first state...\nfirst state:  dict_keys(['observation', 'legal_action', 'reward', 'done', 'model_output_name', 'game_id', 'player_id', 'frame_no', 'sub_action_mask', 'req_pb', 'sgame_id'])\nfirst frame: 0\n----------------------run step  0\n----------------------run step  100\n----------------------run step  200\n----------------------run step  300\n----------------------run step  400\n----------------------run step  500\n----------------------run step  600\n----------------------run step  700\n----------------------run step  800\n----------------------run step  900\n----------------------run step  1000\n----------------------run step  1100\n2023-08-23 13:13:57.782 | INFO     | hok.common.log:info:85 - game not end, send close game at first: cur_frame_no(3525)\n[{\n    \"player_id\": 8,\n    \"frame_no\": 30,\n    \"observation\": array([1.00000000e00, 1.00000000e00, 0.00000000e00, 0.00000000e00, ...]),\n    \"reward\": (0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0),\n    \"legal_action\": array([1.0, 1.0, 1.0, 0.0, 0.0, ...]),\n    \"sub_action_mask\": {\n        0: array([1.0, 0.0, 0.0, 0.0, 0.0, 0.0]),\n        1: array([1.0, 0.0, 0.0, 0.0, 0.0, 0.0]),\n        ...\n        11: array([1.0, 0.0, 0.0, 0.0, 0.0, 0.0]),\n    },\n    \"req_pb\": hok.hok1v1.lib.interface.AIFrameState,\n}, ...]\n```\n\n### Modify 1v1 Game Config\n\nBefore running the game Env, you need to create file `config.json` at the running path.\n\nAn example:\n```json\n{\n  \"reward_money\": \"0.006\",\n  \"reward_exp\": \"0.006\" ,\n  \"reward_hp_point\": \"2.0\",\n  \"reward_ep_rate\": \"0.75\",\n  \"reward_kill\": \"-0.6\",\n  \"reward_dead\": \"-1.0\",\n  \"reward_tower_hp_point\": \"5.0\",\n  \"reward_last_hit\": \"0.5\",\n  \n  \"log_level\": \"4\"\n}\n```\n\nThis config file includes sub-reward factors and log_level of protobuf processing part.\nThe file is only loaded when creating instances of `HoK1v1`,\nand any modifications would not be reloaded even if you call `HoK1v1.reset`.\n\nIn most cases, log_level should be set as `4` to avoid useless log information.\nOnly if you meet some error when using our environment, `log_level` may need a\nlower value to help us get more information about the error your meet.\n\n## 3v3\n\n### Observation and action spaces\n\nPlease refer to [hok3v3](https://doc.aiarena.tencent.com/paper/hok3v3/latest/hok3v3_env/honor-of-kings/) for further information.\n\n### Usage\n\nAssuming you have started your gamecore server at `127.0.0.1:23432` and your IP running the hok_env is `127.0.0.1`.\n\nHere is the basic usage of the hok3v3 environment:\n\n- Get the environment instance:\n\n    ```python\n    GC_SERVER_ADDR = os.getenv(\"GAMECORE_SERVER_ADDR\", \"127.0.0.1:23432\")\n    AI_SERVER_ADDR = os.getenv(\"AI_SERVER_ADDR\", \"127.0.0.1\")\n    reward_config = RewardConfig.default_reward_config.copy()\n    \n    env = get_hok3v3(GC_SERVER_ADDR, AI_SERVER_ADDR, reward_config)\n    ```\n\n- Reset env and start a new game\n\n    ```python\n    use_common_ai = [True, False]\n    camp_config = {\n        \"mode\": \"3v3\",\n        \"heroes\": [\n            [{\"hero_id\": 190}, {\"hero_id\": 173}, {\"hero_id\": 117}],\n            [{\"hero_id\": 141}, {\"hero_id\": 111}, {\"hero_id\": 107}],\n        ],\n    }\n    env.reset(use_common_ai, camp_config, eval_mode=True)\n    ```\n\n- Game loop and predictions\n\n    ```python\n    gameover = False\n    while not gameover:\n        for i, is_comon_ai in enumerate(use_common_ai):\n            if is_comon_ai:\n                continue\n    \n            continue_process, features, frame_state = env.step_feature(i)\n            gameover = frame_state.gameover\n            # only predict every 3 frame\n            if not continue_process:\n                continue\n    \n            probs = random_predict(features, frame_state)\n            ok, results = env.step_action(i, probs, features, frame_state)\n            if not ok:\n                raise Exception(\"step action failed\")\n    \n    env.close_game(force=True)\n    ```\n\nYou can get the default reward config by:\n```python\nreward_config = RewardConfig.default_reward_config.copy()\n```\n\nHere is the [reward config example](aiarena/3v3/actor/config/config.py):\n```python\nreward_config = {\n    \"whether_use_zero_sum_reward\": 1,\n    \"team_spirit\": 0,\n    \"time_scaling_discount\": 1,\n    \"time_scaling_time\": 4500,\n    \"reward_policy\": {\n        \"hero_0\": {\n            \"hp_rate_sqrt_sqrt\": 1,\n            \"money\": 0.001,\n            \"exp\": 0.001,\n            \"tower\": 1,\n            \"killCnt\": 1,\n            \"deadCnt\": -1,\n            \"assistCnt\": 1,\n            \"total_hurt_to_hero\": 0.1,\n            \"atk_monster\": 0.1,\n            \"win_crystal\": 1,\n            \"atk_crystal\": 1,\n        },\n    },\n    \"policy_heroes\": {\n        \"hero_0\": [169, 112, 174],\n    },\n}\n```\n\nYou can get the hero_id by hero_name via [HERO_DICT](hok_env/hok/common/camp.py):\n```python\nfrom hok.common.camp import HERO_DICT\nprint(HERO_DICT)\n```\n\nPlease refer to [hok3v3/test_env](hok_env/hok/hok3v3/unit_test/test_env.py) for the full code introduced above.\n\nAnd you can run the test code by following script:\n\n```shell\npython3.8 -c \"from hok.hok3v3.unit_test.test_env import run_test; run_test()\"\n```\n\nYou will get the following message if works correctly:\n```\n2023-12-28 12:54:28.106 | INFO     | hok.hok3v3.unit_test.test_env:get_hok3v3:14 - Init libprocessor: /usr/local/lib/python3.8/dist-packages/hok/hok3v3/config.dat\n2023-12-28 12:54:28.106 | INFO     | hok.hok3v3.unit_test.test_env:get_hok3v3:15 - Init reward: {'whether_use_zero_sum_reward': 1, 'team_spirit': 0.2, 'time_scaling_discount': 1, 'time_scaling_time': 4500, 'reward_policy': {'policy_name_0': {'hp_rate_sqrt': 1, 'money': 0.001, 'exp': 0.001, 'tower': 1, 'killCnt': 1, 'deadCnt': -1, 'assistCnt': 1, 'total_hurt_to_hero': 0.1, 'ep_rate': 0.1, 'win_crystal': 1}}, 'hero_policy': {1: 'policy_name_0'}, 'policy_heroes': {'policy_name_0': [1, 2]}}\n2023-12-28 12:54:28.107 | INFO     | hok.hok3v3.unit_test.test_env:get_hok3v3:16 - Init gamecore environment: 127.0.0.1:23432 127.0.0.1\n2023-12-28 12:54:28.107 | INFO     | hok.hok3v3.reward:update_reward_config:116 - Update reward config: time_scaling_time:4500, time_scaling_discount:1, team_spirit:0.2, whether_use_zero_sum_reward:1\n2023-12-28 12:54:28.107 | INFO     | hok.hok3v3.reward:update_reward_config:124 - Update hero reward config: 1 -\u003e {'hp_rate_sqrt': 1, 'money': 0.001, 'exp': 0.001, 'tower': 1, 'killCnt': 1, 'deadCnt': -1, 'assistCnt': 1, 'total_hurt_to_hero': 0.1, 'ep_rate': 0.1, 'win_crystal': 1}\n2023-12-28 12:54:28.107 | INFO     | hok.hok3v3.reward:update_reward_config:124 - Update hero reward config: 2 -\u003e {'hp_rate_sqrt': 1, 'money': 0.001, 'exp': 0.001, 'tower': 1, 'killCnt': 1, 'deadCnt': -1, 'assistCnt': 1, 'total_hurt_to_hero': 0.1, 'ep_rate': 0.1, 'win_crystal': 1}\n2023-12-28 12:54:28.136 | INFO     | hok.hok3v3.server:start:31 - Start server at tcp://0.0.0.0:35151\n2023-12-28 12:54:28.139 | INFO     | hok.hok3v3.env:reset:85 - Reset info: agent:0 is_common_ai:True\n2023-12-28 12:54:28.139 | INFO     | hok.hok3v3.env:reset:85 - Reset info: agent:1 is_common_ai:False\n2023-12-28 12:54:30.212 | INFO     | hok.hok3v3.unit_test.test_env:run_test:78 - ----------------------run step 0\n2023-12-28 12:54:30.673 | INFO     | hok.hok3v3.unit_test.test_env:run_test:78 - ----------------------run step 100\n2023-12-28 12:54:30.945 | INFO     | hok.hok3v3.unit_test.test_env:run_test:78 - ----------------------run step 200\n```\n\n## Cluster training\n\nPlease consult [cluster.md](docs/cluster.md) document for instructions on cluster training utilizing the `hok_env` environment and the integrated `rl_framework`.\n\n## Replay software: ABS Parsing Tool (will be provided along with the gamecore)\n\nWatching the game is a direct way to see the performance of your agent throughout a match. We provide a replay tool to visualize the matches.\n\nThis is an official replay software which parses the ABS file generated by the gamecore and outputs the videos in the game UI of Honor of Kings.\nThe ABS file generated by the gamecore could be found under the gamecore folder.\nYou can visualize the matches by putting  ABS files under the `Replays` folder and running **ABSTOOL.exe**.\n![avatar](./docs/replay-tool.gif)\n\n## Pre-built image\n\nhttps://hub.docker.com/r/tencentailab/hok_env\n\nSee also: [run_with_prebuilt_image](./docs/run_with_prebuilt_image.md)\n\n## Install from PyPI\n\n```\npip install hok\n```\n\n## Citation\n\nIf you use the gamecore of hok_env or the code in this repository, please cite our paper as follows.\n\n```\n@inproceedings{wei2022hok_env,\n  title={Honor of Kings Arena: an Environment for Generalization in Competitive Reinforcement Learning},\n  author={Wei, Hua and Chen, Jingxiao and Ji, Xiyang and Qin, Hongyang and Deng, Minwen and Li, Siqin and Wang, Liang and Zhang, Weinan and Yu, Yong and Liu, Lin and Huang, Lanxiao and Ye, Deheng and Fu, Qiang and Yang, Wei},\n  booktitle={Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks},\n  year={2022}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftencent-ailab%2Fhok_env","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftencent-ailab%2Fhok_env","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftencent-ailab%2Fhok_env/lists"}