{"id":23897679,"url":"https://github.com/omerkel/ucthello","last_synced_at":"2025-04-10T17:05:09.618Z","repository":{"id":71262995,"uuid":"51536241","full_name":"OMerkel/UCThello","owner":"OMerkel","description":"UCThello - a board game demonstrator (Othello variant) with computer AI using Monte Carlo Tree Search (MCTS) with UCB (Upper Confidence Bounds) applied to trees (UCT in short)","archived":false,"fork":false,"pushed_at":"2023-07-10T16:35:08.000Z","size":5886,"stargazers_count":25,"open_issues_count":0,"forks_count":4,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-24T14:46:08.471Z","etag":null,"topics":["2-player-strategy-game","abstract-game","ai","ai-players","artificial-intelligence","board-game","entertainment","game","mcts","mobile","mobile-app","mobile-game","monte-carlo-tree-search","othello","perfect-information","simulation","ucb","uct","upper-confidence-bounds"],"latest_commit_sha":null,"homepage":"http://omerkel.github.io/UCThello/html5/src","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OMerkel.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2016-02-11T18:24:57.000Z","updated_at":"2024-11-16T00:12:49.000Z","dependencies_parsed_at":"2025-04-10T17:03:39.797Z","dependency_job_id":null,"html_url":"https://github.com/OMerkel/UCThello","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OMerkel%2FUCThello","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OMerkel%2FUCThello/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OMerkel%2FUCThello/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OMerkel%2FUCThello/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OMerkel","download_url":"https://codeload.github.com/OMerkel/UCThello/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248260123,"owners_count":21074207,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["2-player-strategy-game","abstract-game","ai","ai-players","artificial-intelligence","board-game","entertainment","game","mcts","mobile","mobile-app","mobile-game","monte-carlo-tree-search","othello","perfect-information","simulation","ucb","uct","upper-confidence-bounds"],"created_at":"2025-01-04T17:17:11.863Z","updated_at":"2025-04-10T17:05:09.606Z","avatar_url":"https://github.com/OMerkel.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg alt=\"UCThello icon\" width=\"64\" src=\"html5/src/img/icons/ucthello64.png\" /\u003e UCThello\n====================\n\n* \u003cem\u003eStart an online\u003c/em\u003e \u003cb\u003eUCThello\u003c/b\u003e \u003cem\u003esession on\u003c/em\u003e http://omerkel.github.io/UCThello/html5/src\n* \u003cem\u003eAndroid APK available for install\u003c/em\u003e \u003cimg align=\"top\" width=\"32\" src=\"res/android.gif\" /\u003e https://github.com/OMerkel/UCThello/releases\n* \u003cem\u003eRuns in various browsers on\u003c/em\u003e\n    * \u003cem\u003edesktop systems like BSDs, Linux, Win, MacOS and\u003c/em\u003e\n    * \u003cem\u003emobile platforms like Android, FirefoxOS, iOS.\u003c/em\u003e\n\n_UCThello - a board game demonstrator with computer AI using\nMonte-Carlo Tree Search (MCTS) with UCB (Upper Confidence Bounds)\napplied to trees (UCT in short)_\n\n__Keywords, Categories__ _Monte-Carlo Tree Search (MCTS),\nUpper Confidence Bounds (UCB), UCB applied to trees (UCT), AI,\n2-player board game, deterministic game with perfect information,\nJavaScript, ECMAScript, W3C WebWorker_ \n\n# Abstract\n\nUCThello is a board game using Monte-Carlo Tree Search (MCTS) with\nUCB (Upper Confidence Bounds) applied to trees (UCT in short) for the\ncomputer player AI. The board game used for demonstration purposes of\nthe UCT algorithm is close to a game named _Othello_ depending on\nselected options. In fact it can be played depending on your\nconfiguration following the official tournament rules of the\nWOF - World Othello Federation - if intended [WOF14]. Other rule settings\nto play variants are available, too.\nPer design decision the playing strength is limited for\npleasure and fun level. Thus it is kept at a moderate to quite\nstrong level on purpose due to the target environment, device\nplatform, and audience expectations.\nThis is done e.g. by limitation of the maximum AI response time and\nusing a single execution thread for AI only plus just a\nsecond independent execution thread for a responsive user\ninterface to avoid battery drains if full CPU and GPU\ncore support would be implemented leading to bad user experience.\nOther possible but at least currently postponed improvements could\nbe done by simple usage of a well-known and available game opening book.\nAlthough such simple modifications could improve the playing strength\nthese features are not implemented in the current version yet.\n\n_Othello_ is a derivative of the board game _Reversi_ which can be\nplayed by UCThello as well. _Reversi_ is claimed to be invented by\neither Lewis Waterman or John W. Mollett. Predecessor of _Reversi_ created\nby Mollett is _The game of Annexation_, also called _Annex_ back in\n19th century.\n\n# Monte-Carlo Tree Search\n\nThe __Monte-Carlo Tree Search__ (MCTS in short) represents an algorithms used to build a\n_Search Tree_ interatively by successively adding nodes according to traversing of\nnodes and simulations in the problem domain. If the problem domain is a game then\nthe nodes can represent moves according to the game rules.\nTraversing nodes follows a _Selection Strategy_. _Simulations_ are often called\n_playouts_, too. The different nodes inside the simulated paths get statistics\nreflecting ratios of win and loss related to total amount of simulations.\nAssumption is that with higher total amount of simulations the confidence in the\nstatistics gets high enough and allows to select quality nodes or moves.\nSuch that the idea is to retrieve the acceptable next node or move with optimal\nratio then.\n\n\u003cimg src='http://omerkel.github.io/UCThello/res/mcts_1_typical_mcts_states.svg' /\u003e\n\nThe iterative MCTS algorithm is modelled to perform four main states typically called\n* _Selection_,\n* _Expansion_,\n* _Simulation_, and\n* _Backpropagation_. See [Cha10] \u0026amp; [CBSS08]\n\nIn UCThello the related code fragment for this loop is close to\n\n```\nUct.prototype.getActionInfo = function ( board, maxIterations, verbose ) {\n  var root = new UctNode(null, board, null);\n  for(var iterations=0; iterations\u003cmaxIterations; iterations+=1) {\n    var node = root;\n    var variantBoard = board.copy();\n    /* Selection */\n    ...\n    /* Expansion */\n    ...\n    /* Simulation */\n    ...\n    /* Backpropagation */\n    ...\n  }\n  return { action : root.mostVisitedChild().action };\n};\n```\n\nOn a given board the most visited and therefore best information describing an action\naccording to the rules performed by the current player shall be determined.\n\n## Selection\n\nFirst step or state in an MCTS algorithm iteration is the __Selection__. _Objective\nof the Selection_ is to retrieve a path beginning at the root node towards a selected\nleaf node from the search tree. The Search Tree stays fixed inside the Selection state. It\ngrows in a later state of the algorithm by appending more nodes on each iteration of\nthe MCTS. Only exception is when a selected path has a final leaf node that is a\n_terminal node_. A terminal node simply is a move representation of an end of game situation\naccording to the rules. The root node represents the current game or problem domain\nsituation. To traverse the search tree from the root node towards the leaf nodes\nsimply means to follow a possible predicted _variant_ of game play.\n\n\u003cimg src='http://omerkel.github.io/UCThello/res/mcts_2_selection.svg' /\u003e\n\nThe _objective of the Selection Strategy_ is to branch the intended search path in\na balance of information _exploration_ and _exploitation_. If a branch is selected\nfollowing a search path branch already examined previously this is seen as an\nexploit. An exploit shall confirm the quality of an already examined node in terms\nof gaining higher statistical confidence. Higher statistical confidence does mean\nto have more reliable estimates. Exploration is performed by creating new\nnodes in later MCTS steps or alternatively search path branch selection of\nrelatively rare traversed nodes. Nodes traversed in a low amount simply\nreflects a low reliability or statistical confidence. The border between\nexploit and explore is often seen as being soft and fluent.\n\nThus a selection of a child node to traverse next at each level of the already\nbuild search tree path is usually based on a quality value of the visited\nnodes in earlier iterations.\nAn optimal _Selection Strategy_ to best support the objective is unknown. One\nstatistical approach called _Upper Confidence Bounds_ (UCB) algorithm uses\na logarithm based formula on collected quality values correlated to the\nnodes on the search path if applied to MCTS. The combination of MCTS and UCB\ncalled UCT (short for _UCB applied to trees_) is credited to [KS06]. Other\napproaches or additional supporting ideas for a _Selection Strategy_ are\npresented and discussed e.g. in [CSUB06].\n\nBesides the Selection Strategy in search path branch Selection an additional\naspect is seen. To avoid a risk that any high quality node is unvisited that\nis located near the rood node already. To reach such a design goal a possible\nsolution is to favor traversing any unexplored child node over following\nexplored siblings. Widening the search tree is then favored over deepening.\nCritics could be that randomness of Monte-Carlo methods is reduced if applied.\n\nIn UCThello the select child step implements the UCT algorithm. The UCB\nrelated code is part of the _UctNode.prototype.selectChild_ function.\n\nAdditionally UCThello implements to favor early _Selection_ of a traversed\nnode on any unexplored (or unexamined) child existing. Such an unexplored\n(or unexamined) child is preferred over continuing traversing any\nexplored node.\n\n```\nvar node = root;\nvar variantBoard = board.copy();\n/* Selection */\nwhile (node.unexamined.length == 0 \u0026\u0026 node.children.length \u003e 0) {\n  node = node.selectChild();\n  variantBoard.doAction(node.action);\n}\n```\n\n## Expansion\n\nThe objective of the __Expansion__ step is to add a new unexplored child of\nthe node determined by the previous _Selection_.\n\nIf the node determined by the _Selection_ is an inner node instead of a\nleaf node then this node has a combination of explored and unexplored\nchildren. Either way an unexplored child shall be added for the\ncoming _Simulation_ state. Only exception is that a leaf node has been\nreached representing a terminal node. In such a case no Expansion and\nSimulation is needed since a terminal node means that a end of game\nis implied at that node on the search path.\n\nSometimes you will find implementations where multiple Expansions take\nplace on the Selection node. This simply means a set of child nodes is\nadded at once then.\n\nIn UCThello exactly one node will be added unless a terminal node is\nreached and the list of remaining unexplored child nodes is\ndetermined before. To avoid any preferred order when getting a node\nfrom the set of remaining nodes or when a dependency from any\nparameter or state exists the returned node is selected randomly.\n\n```\n/* Selection */\n...\n/* Expansion */\nif (node.unexamined.length \u003e 0) {\n  var j = Math.floor(Math.random() * node.unexamined.length);\n  variantBoard.doAction(node.unexamined[j]);\n  node = node.addChild(variantBoard, j);\n}\n```\n\nTerminal nodes do not have any child nodes. So it is sufficient to\ncheck for the unexamined.length in case a terminal node has been\nselected.\n\n## Simulation\n\nNow the objective of a __Simulation__ is to playout a possible scenario\nstarting from the newly expanded search tree leaf node. Simulation is\nperformed until end of game is reached.\n\nMind the playout does not modify the expanded search tree leaf node. The fixed\nleaf node - respectively the correlated game state - is used as the base for\nthe simulation only.\n\nOn each simulation step a player's action valid by the rules is performed on the\ncreated variant board. The variant board is used as a complete copy of the\ncurrent board and game state. This is to avoid changes to the board and game\nstate while following the full search path and simulation steps.\n\nInstead of doing just a single playout alternatively several playouts could be\nstarted from the selected and expanded search tree leaf node. Idea behind\nthis would be to save the run time needed for a possible choice of the same\nselection path in later iterations.\n\nIn UCThello a single playout is performed per iteration. The number of MCTS\nalgorithm iterations equals the number of simulations then.\n\n```\nvar variantBoard = board.copy();\n/* Selection */\n...\n/* Expansion */\n...\n/* Simulation */\nvar actions = variantBoard.getActions();\nwhile(actions.length \u003e 0) {\n  variantBoard.doAction(actions[Math.floor(Math.random() * actions.length)]);\n  ...\n  actions = variantBoard.getActions();\n}\n```\n\n## Backpropagation\n\nObjective of the __Backpropagation__ is to update the statistics of all nodes\nalong the search tree path in reverse order until the root node is reached.\nThe Simulation did not perform any changes on the search tree path. Since the\nsearch tree path is unchanged this means the eventually played or predicted\nresult on the playout can be used to update statistics starting at the\nsearch tree path leaf node via the parent nodes until the root node is\nreached.\n\n```\nvar node = root;\nvar variantBoard = board.copy();\n/* Selection */\n...\n/* Expansion */\n...\n/* Simulation */\n...\n/* Backpropagation */\nvar result = variantBoard.getResult();\nwhile(node) {\n  node.update(result);\n  node = node.parentNode;\n}\n```\n\nIn UCThello the UCT AI player does not maximize for the\namount of discs of own color on board.\nInstead it analyzes the end of game situation just for any\nresult being a win. The call variantBoard.getResult()\nreturns an array of length two.\nThe two values returned stand for the game result of \nthe players in terms of win or loss. The winning player\ngets a full point while his opponent scores zero points.\nMeaning the result is either [ 1, 0 ] or [ 0, 1 ].\nA draw or stalemate situation is represented as an array\n[ 0.5, 0.5 ]. Meaning a draw is better than a loss but\nshall be interpreted as half a win for both players.\n\nThe statistics for a node is updated by node.update(result).\nMind the search tree node is representing a move of the active\nplayer according to the rules. The update picks the active\nplayer's end of game result from the result array and adds it\nto a statistics value representing the total amount of wins\nfound traversing the search tree node over all MCTS iterations.\nAdditionally the amount of visits for the node is increased.\n\n# References\n\n* __[Cha10]__ Guillaume Maurice Jean-Bernard Chaslot, \"[Monte-Carlo Tree Search](https://project.dke.maastrichtuniversity.nl/games/files/phd/Chaslot_thesis.pdf)\", PHD Proefschrift, Universiteit Maastricht, NL, 2010.\n* __[CBSS08]__ Guillaume Chaslot, Sander Bakkes, Istvan Szita and Pieter Spronck, \"[Monte-Carlo Tree Search: A New Framework for Game AI](http://sander.landofsand.com/publications/AIIDE08_Chaslot.pdf)\", in Proceedings of the Fourth Artificial Intelligence and Interactive Digital Entertainment Conference, Stanford, California, 2008. Published by The AAAI Press, Menlo Park, California.\n* __[KS06]__ Levente Kocsis, Csaba Szepesvári, \"[Bandit based Monte-Carlo Planning](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.102.1296)\", In European Conference on Machine Learning (ECML) 2006, Lecture Notes in Artificial Intelligence 4212, pp. 282–293, 2006.\n* __[CSUB06]__ Guillaume Chaslot, Jahn-Takeshi Saito, Jos W.H.M. Uiterwijk, Bruno Bouzy, H. Jaap van den Herik, \"[Monte-Carlo Strategies for Computer Go](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.97.8924)\", In Proceedings of the 18th Belgian-Dutch Conference on Artificial Intelligence, pp. 83–90, 2006.\n* Brian Rose, \"[Othello. A Minute to Learn... A Lifetime to Master](http://www.ffothello.org/livres/othello-book-Brian-Rose.pdf)\", 2005.\n* __[WOF14]__ World Othello Federation, \"[World Othello Championship Rules](http://www.worldothello.nu/sites/default/files/field/image/wocrules2014.pdf)\", as valid for the 39th World Othello Championship 2015, Cambridge, UK, October 2015.\n\n# 3rd Party Libraries\n\n* jQuery: MIT licensed, https://github.com/jquery/jquery\n* jQuery Mobile: MIT licensed, https://github.com/jquery/jquery-mobile\n\n# Links\n\n* Association for the Advancement of Artificial Intelligence, http://www.aaai.org\n* HTML Living Standard, Web Workers, https://html.spec.whatwg.org\n* The Othello Museum, http://www.beppi.it/public/OthelloMuseum/pages/history.php\n\n## Othello Organizations\nMind that UCThello follows (most) official tournament rules of the listed\norganizations depending on your selected options. Still UCThello is\nindependent development from any work of these organizations.\n\n* World Othello Federation, http://www.worldothello.org\n* Australian Othello Federation, http://www.othello.asn.au\n* British Othello Federation, http://www.britishothello.org.uk\n* Dansk Othello Forbund, http://www.othello.dk\n* Fédération Française d’Othello, http://www.ffothello.org \n* Federazione Nazionale Gioco Othello, Italia, http://www.fngo.it\n* Malaysia Othello Association (MOA), http://z12.invisionfree.com/MOA/index.php\n* Othello Club Deutschland, http://www.othello-club.de.vu\n* United States Othello Association (USOA), http://www.usothello.org\n\n# Contributors / Authors\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e\u003cp\u003eOliver Merkel,\u003cbr /\u003e\u003ca rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-nd/4.0/\"\u003e\u003cimg alt=\"Creative Commons License\" style=\"border-width:0\" src=\"http://i.creativecommons.org/l/by-nc-nd/4.0/88x31.png\" /\u003e\u003c/a\u003e\u003cbr /\u003eThis image is licensed under a \u003ca rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-nd/4.0/\"\u003eCreative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License\u003c/a\u003e.\n    \u003c/p\u003e\n    \u003c/td\u003e\n    \u003ctd width=\"30%\"\u003e\u003cimg width=\"100%\" ondragstart=\"return false;\" alt=\"Oliver Merkel, Creative Commons License, This image is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.\" src=\"html5/src/img/oliver_moher-150812.jpg\" /\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n_All logos, brands and trademarks mentioned belong to their respective owners._\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fomerkel%2Fucthello","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fomerkel%2Fucthello","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fomerkel%2Fucthello/lists"}