{"id":23933783,"url":"https://github.com/masouduut94/MCTS-agent-python","last_synced_at":"2025-09-11T16:32:49.025Z","repository":{"id":148411003,"uuid":"214206049","full_name":"masouduut94/MCTS-agent-python","owner":"masouduut94","description":"Monte Carlo Tree Search (MCTS) is a method for finding optimal decisions in a given domain by taking random samples in the decision space and building a search tree accordingly. It has already had a profound impact on Artificial Intelligence (AI) approaches for domains that can be represented as trees of sequential decisions, particularly games and planning problems. In this project I used a board game called \"HEX\" as a platform to test different simulation strategies in MCTS field.","archived":false,"fork":false,"pushed_at":"2024-03-10T00:26:42.000Z","size":712,"stargazers_count":58,"open_issues_count":1,"forks_count":8,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-03-10T01:32:02.686Z","etag":null,"topics":["decision-space","game-of-hex","markov-decision-processes","mcts","monte-carlo-tree-search","reinforcement-learning","sequential-decisions"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/masouduut94.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-10-10T14:35:31.000Z","updated_at":"2024-02-17T21:38:44.000Z","dependencies_parsed_at":null,"dependency_job_id":"c7752f1f-2528-477a-bcad-658b09cf05d3","html_url":"https://github.com/masouduut94/MCTS-agent-python","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/masouduut94%2FMCTS-agent-python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/masouduut94%2FMCTS-agent-python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/masouduut94%2FMCTS-agent-python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/masouduut94%2FMCTS-agent-python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/masouduut94","download_url":"https://codeload.github.com/masouduut94/MCTS-agent-python/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":232658691,"owners_count":18556988,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["decision-space","game-of-hex","markov-decision-processes","mcts","monte-carlo-tree-search","reinforcement-learning","sequential-decisions"],"created_at":"2025-01-06T00:29:54.638Z","updated_at":"2025-01-06T00:33:59.350Z","avatar_url":"https://github.com/masouduut94.png","language":"Python","funding_links":[],"categories":["Building"],"sub_categories":["Testing"],"readme":"# HexPy\n\n### Monte Carlo Tree Search Agent for the game of HEX\n\n## Demo:\n\n![Demo of MCTS General Game Player](https://github.com/masouduut94/MCTS-agent-python/blob/master/resources/demo.gif) \n\n## Description\nThis code belongs to this paper **:link: [IMPROVING MONTE CARLO TREE SEARCH BY COMBINING\nRAVE AND QUALITY-BASED REWARDS ALGORITHMS](https://github.com/masouduut94/MCTS-agent-python/blob/master/paper/CONFITC04_172.pdf)**.\n\n### what is Monte Carlo Tree Search(MCTS)?\nMONTE Carlo Tree Search (MCTS) is a method for finding optimal decisions in a given domain by\ntaking random samples in the decision space and building a search tree according to the results.\nIt has already had a profound impact on Artificial Intelligence (AI) approaches for domains that\ncan be represented as trees of sequential decisions, particularly games and planning problems. \nIn this project I used different simulation strategies to enhance the agent policy to explore the environment.\n\n from :link: [A Survey of Monte Carlo Tree Search Methods](http://ieeexplore.ieee.org/abstract/document/6145622/)\n\n### About contribution\nBefore you go through the details, I recommend you to get familiar with the framework reading these medium articles:\n- [A simple no math included introduction to reinforcement learning](https://towardsdatascience.com/monte-carlo-tree-search-a-case-study-along-with-implementation-part-1-ebc7753a5a3b)\n- [A simple introduction to Monte Carlo Tree Search](https://towardsdatascience.com/monte-carlo-tree-search-implementing-reinforcement-learning-in-real-time-game-player-25b6f6ac3b43)\n- [Details of MCTS implementation on game of HEX](https://towardsdatascience.com/monte-carlo-tree-search-implementing-reinforcement-learning-in-real-time-game-player-a9c412ebeff5)\n\nSo if you are familiar with the whole concept of MCTS and UCT algorithm, you must know that in practice it suffers from \nsparse rewards. it takes so much time to warm up the tree with simple UCT algorithm. So in this case we **first implemented\nthe RAVE algorithm** that helps warm up tree faster. then implemented **several simulation strategy like Last Good Reply,\nPoolRAVE, Decisive Move and also UCB1-Tuned**.\n\nThen we applied **quality based rewards** in [Quality-based Rewards for Monte-Carlo Tree Search Simulations](https://dl.acm.org/doi/10.5555/3006652.3006771) \nwhich basically it asserts that we can apply discounted rewards by **knowing the length of simulation and the \nmaximum number of actions allowed to take in environment** for each player (In some games, the game ends after limited number of moves. because there is no more movements).\n\nAfter that we used **HRAVE and GRAVE in the paper [Comparison of rapid action value estimation variants for general game playing 2018 - Chiara F. Sironi; Mark H. M. Winands](https://ieeexplore.ieee.org/document/7860429)**\nwhich basically states that we can use the **global information of the game to guide the simulations**.\nWe also tested the **leaf threading on UCT**.\n\nall of above algorithms are addressed below.\n\n### Original repo\n\n- MopyHex: Authored by Kenny Young [here](https://github.com/kenjyoung/mopyhex)\n\n### Contributions to the original repo:\n- implementing Generalized Rapid Action Value Estimation\n- implementing HRAVE and GRAVE algorithms in [Comparison of rapid action value estimation variants for general game playing 2018 - Chiara F. Sironi; Mark H. M. Winands](https://ieeexplore.ieee.org/document/7860429)\n- implementing Quality-based rewards in [Quality-based Rewards for Monte-Carlo Tree Search Simulations](https://dl.acm.org/doi/10.5555/3006652.3006771)\n- implementing leaf-threading on basic No frills UCT.\n\n**This project has a further optimized version in [here](https://github.com/masouduut94/MCTS-agent-cythonized) which optimized by cython.**\n\n### Researches have been done in **Urmia University of Technology**.\n\u003cp align=\"center\"\u003e\n\n\u003cimg src=\"https://github.com/masouduut94/MCTS-agent-python/blob/master/image/ssss.png\"\u003e\n    \n\u003c/p\u003e\n\n#### Authors: \n- Masoud Masoumi Moghadam (Me :sunglasses:)\n- Prof: Mohammad Pourmahmood Aghababa [profile](https://bit.ly/3dV23Be)\n- Prof: Jamshid Bagherzadeh [profile](https://bit.ly/3dPX4Sc)\n\n## What is monte carlo tree search anyway?\n\n\n# Requirements\n- OS: Windows  and Ubuntu\n- tkinter\n- Numpy\n\n# To run it:\nYou can :running: (run) program using this command:\n\n    python main.py\n\nAlso you can run tests for comparing two mcts-based algorithms against \neach other using the `playtest.py`.\n\n## :closed_book: To know more about MCTS:\n\nThis one is highly recommended: \n\n\n## Algorithms used for boosting MCTS in this framework: \n\n- Upper Confidence Bounds (UCT)\n- UCB1-Tuned\n- Rapid Action Value Estimation (RAVE)\n- Decisive Move \n- Quality Based Rewards\n- Pool RAVE\n- Last Good Reply\n\n\n# References\n- [1] A Survey of Monte Carlo Tree Search Methods, Cameron B. Browne et al, 2012 [Link to paper](https://ieeexplore.ieee.org/document/6145622)\n- [2] Generalized Rapid Action Value Estimation, Tristan Cazenave,  2017 [Link to paper](https://www.ijcai.org/Proceedings/15/Papers/112.pdf)\n- [3] Comparison of rapid action value estimation variants for general game playing, C. Sironi, 2018 [Link to paper](https://ieeexplore.ieee.org/document/7860429)\n- [4] Quality-based Rewards for Monte-Carlo Tree Search Simulations, 2014 [Link to paper](https://dl.acm.org/doi/10.5555/3006652.3006771)\n- [5] The Last-Good-Reply Policy for Monte-Carlo Go 2009 [Link to paper](https://www.semanticscholar.org/paper/The-Last-Good-Reply-Policy-for-Monte-Carlo-Go-Drake/980e6b8ef765b0fe4fc3fe8f068c79ac4169b00f) \n- [6] On the Huge Benefit of Decisive Moves in Monte-Carlo Tree Search Algorithms, Fabien Teytaud, Olivier Teytaud, 2010\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmasouduut94%2FMCTS-agent-python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmasouduut94%2FMCTS-agent-python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmasouduut94%2FMCTS-agent-python/lists"}