{"id":23080746,"url":"https://github.com/chen0040/java-reinforcement-learning-tic-tac-toe","last_synced_at":"2025-07-17T00:32:23.646Z","repository":{"id":85884481,"uuid":"112411562","full_name":"chen0040/java-reinforcement-learning-tic-tac-toe","owner":"chen0040","description":"Demo of reinforcement learning using tic-tac-toe","archived":false,"fork":false,"pushed_at":"2017-12-18T02:44:18.000Z","size":32,"stargazers_count":8,"open_issues_count":0,"forks_count":6,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-01T20:02:05.123Z","etag":null,"topics":["q-learning","reinforcement-learning","sarsa","tic-tac-toe"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chen0040.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-11-29T01:42:53.000Z","updated_at":"2025-01-13T15:42:29.000Z","dependencies_parsed_at":null,"dependency_job_id":"6d7c2229-90fa-43bb-adf0-4dfd498bf648","html_url":"https://github.com/chen0040/java-reinforcement-learning-tic-tac-toe","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/chen0040/java-reinforcement-learning-tic-tac-toe","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chen0040%2Fjava-reinforcement-learning-tic-tac-toe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chen0040%2Fjava-reinforcement-learning-tic-tac-toe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chen0040%2Fjava-reinforcement-learning-tic-tac-toe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chen0040%2Fjava-reinforcement-learning-tic-tac-toe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chen0040","download_url":"https://codeload.github.com/chen0040/java-reinforcement-learning-tic-tac-toe/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chen0040%2Fjava-reinforcement-learning-tic-tac-toe/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265554787,"owners_count":23787279,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["q-learning","reinforcement-learning","sarsa","tic-tac-toe"],"created_at":"2024-12-16T13:16:18.158Z","updated_at":"2025-07-17T00:32:23.637Z","avatar_url":"https://github.com/chen0040.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# java-reinforcement-learning-tic-tac-toe\n\nDemo of reinforcement learning using tic-tac-toe with the [java-reinforcement-learning](https://github.com/chen0040/java-reinforcement-learning) package.\n\n# Usage - GUI\n\nThe gui version run reinforcement learning in the interactive mode. it is implemented in com.github.chen0040.jrl.ttt.gui.Application\n\nTo run the gui version, git clone this project, run the make.ps1 or make.sh to build the reinforcement-learning-tic-tac-toe.jar into the bin folder.\n\nRun:\n\n```bash\njava -jar bin/reinforcement-learning-tic-tac-toe.jar\n```\n\n# Usage - Console\n\nThe console version runs the reinforcement learning against a naive bot, they are implemented in:\n\n* com.github.chen0040.jrl.ttt.dojos.DojoQ\n* com.github.chen0040.jrl.ttt.dojos.DojoSarsa\n* com.github.chen0040.jrl.ttt.dojos.DojoR\n\n### Q-Learn \n\nThe following create two Q-Learning bots that plays against each other on the tic-tac-toe game to a train a Q-Learn model:\n\n```java\npublic static QLearner train(Board board, int episodes) {\n\n    int stateCount = (int)Math.pow(3, board.size() * board.size());\n    int actionCount = board.size() * board.size();\n\n    QLearner learner = new QLearner(stateCount, actionCount);\n    //learner.setActionSelection(SoftMaxActionSelectionStrategy.class.getCanonicalName());\n\n    QBot bot1 = new QBot(1, board, learner);\n    QBot bot2 =new QBot(2, board, learner);\n    \n    int wins = 0;\n\n    for(int i=0; i \u003c episodes; ++i) {\n        bot1.clearHistory();\n        bot2.clearHistory();\n\n        logger.info(\"Iteration: {} / {}\", (i+1), episodes);\n        board.reset();\n        while (board.canBePlayed()) {\n            bot1.act();\n            bot2.act();\n        }\n        logger.info(\"winner: {}\", board.getWinner());\n        bot1.updateStrategy();\n        bot2.updateStrategy();\n        logger.info(\"board: \\n{}\", board);\n        \n        wins += board.getWinner() == 1 ? 1 : 0;\n        logger.info(\"success rate: {} %\", (wins * 100) / (i+1));\n    }\n\n    return learner;\n}\n```\n\nAlternaitvely, the following create one QBot that plays against a NaiveBot on the tic-tac-toe game to a train a Q-Learn model:\n\n```java\npublic static QLearner train(Board board, int episodes) {\n\n    int stateCount = (int)Math.pow(3, board.size() * board.size());\n    int actionCount = board.size() * board.size();\n\n    QLearner learner = new QLearner(stateCount, actionCount);\n    //learner.setActionSelection(SoftMaxActionSelectionStrategy.class.getCanonicalName());\n\n    QBot bot1 = new SarsaBot(1, board, learner);\n    NaiveBot bot2 =new SarsaBot(2, board);\n\n    for(int i=0; i \u003c episodes; ++i) {\n        bot1.clearHistory();\n        bot2.clearHistory();\n\n        logger.info(\"Iteration: {} / {}\", (i+1), episodes);\n        board.reset();\n        while (board.canBePlayed()) {\n            bot1.act();\n            bot2.act();\n        }\n        logger.info(\"winner: {}\", board.getWinner());\n        bot1.updateStrategy();\n        logger.info(\"board: \\n{}\", board);\n    }\n\n    return learner;\n}\n```\n\nThe following code uses the trained Q-Learn model to create a QBot (bot1) which plays against bot2 (which is a naive bot that randomly takes a possible action each step):\n\n```java\npublic static double test(Board board, QLearner model, int episodes) {\n\n    QBot bot1 = new QBot(1, board, model);\n    NaiveBot bot2 =new NaiveBot(2, board);\n\n    int wins = 0;\n    int loses = 0;\n    for(int i=0; i \u003c episodes; ++i) {\n        bot1.clearHistory();\n        bot2.clearHistory();\n\n        logger.info(\"Iteration: {} / {}\", (i+1), episodes);\n        board.reset();\n        while (board.canBePlayed()) {\n            bot1.act();\n            bot2.act();\n        }\n        int winner = board.getWinner();\n        logger.info(\"Winner: {}\", winner);\n        wins += winner == 1 ? 1 : 0;\n        loses += winner == 2 ? 1 : 0;\n    }\n\n    return wins * 1.0 / episodes;\n}\n```\n\nThe following code shows how to train and then test a Q-Learn model:\n\n```java\nBoard board = new Board();\nQLearner model = train(board, 30000);\n\ndouble successRate = test(board, model, 1000);\nlogger.info(\"Q-Learn Bot beats Random Bot in {} % of the games being played\", successRate * 100);\n```\n\nTo save and load the Q-Learn model:\n\n```java\nString model_json = model.toJson();\nQLearner loaded_from_json = QLearner.fromJson(model_json);\n```\n\nThis sample code can be found in the DojoQ.java file in the project.\n\n### SARSA (State-Action-Reward-State-Action)\n\nThe following create two SARSA bots that plays against each other on the tic-tac-toe game to a train a SARSA model:\n\n```java\npublic static QLearner train(Board board, int episodes) {\n\n    int stateCount = (int)Math.pow(3, board.size() * board.size());\n    int actionCount = board.size() * board.size();\n\n    SarsaLearner learner = new SarsaLearner(stateCount, actionCount);\n    //learner.setActionSelection(SoftMaxActionSelectionStrategy.class.getCanonicalName());\n\n    SarsaBot bot1 = new SarsaBot(1, board, learner);\n    SarsaBot bot2 =new SarsaBot(2, board, learner);\n\n    for(int i=0; i \u003c episodes; ++i) {\n        bot1.clearHistory();\n        bot2.clearHistory();\n\n        logger.info(\"Iteration: {} / {}\", (i+1), episodes);\n        board.reset();\n        while (board.canBePlayed()) {\n            bot1.act();\n            bot2.act();\n        }\n        logger.info(\"winner: {}\", board.getWinner());\n        bot1.updateStrategy();\n        bot2.updateStrategy();\n        logger.info(\"board: \\n{}\", board);\n    }\n\n    return learner;\n}\n```\n\nAlternaitvely, the following create one SARSA bot that plays against a NaiveBot on the tic-tac-toe game to a train a SARSA model:\n\n```java\npublic static QLearner train(Board board, int episodes) {\n\n    int stateCount = (int)Math.pow(3, board.size() * board.size());\n    int actionCount = board.size() * board.size();\n\n    SarsaLearner learner = new SarsaLearner(stateCount, actionCount);\n    //learner.setActionSelection(SoftMaxActionSelectionStrategy.class.getCanonicalName());\n\n    SarsaBot bot1 = new SarsaBot(1, board, learner);\n    NaiveBot bot2 =new SarsaBot(2, board);\n    \n    int wins = 0;\n\n    for(int i=0; i \u003c episodes; ++i) {\n        bot1.clearHistory();\n        bot2.clearHistory();\n\n        logger.info(\"Iteration: {} / {}\", (i+1), episodes);\n        board.reset();\n        while (board.canBePlayed()) {\n            bot1.act();\n            bot2.act();\n        }\n        logger.info(\"winner: {}\", board.getWinner());\n        bot1.updateStrategy();\n        logger.info(\"board: \\n{}\", board);\n        \n        wins += board.getWinner() == 1 ? 1 : 0;\n        logger.info(\"success rate: {} %\", (wins * 100) / (i+1));\n    }\n\n    return learner;\n}\n```\n\nThe following code uses the trained SARSA model to create a SarsaBot (bot1) which plays against bot2 (which is a naive bot that randomly takes a possible action each step):\n\n```java\npublic static double test(Board board, SarsaLearner model, int episodes) {\n\n    SarsaBot bot1 = new SarsaBot(1, board, model);\n    NaiveBot bot2 =new NaiveBot(2, board);\n\n    int wins = 0;\n    int loses = 0;\n    for(int i=0; i \u003c episodes; ++i) {\n        bot1.clearHistory();\n        bot2.clearHistory();\n\n        logger.info(\"Iteration: {} / {}\", (i+1), episodes);\n        board.reset();\n        while (board.canBePlayed()) {\n            bot1.act();\n            bot2.act();\n        }\n        int winner = board.getWinner();\n        logger.info(\"Winner: {}\", winner);\n        wins += winner == 1 ? 1 : 0;\n        loses += winner == 2 ? 1 : 0;\n    }\n\n    return wins * 1.0 / episodes;\n}\n```\n\nThe following code shows how to train and then test a SARSA model:\n\n```java\nBoard board = new Board();\nSarsaLearner model = train(board, 30000);\n\ndouble successRate = test(board, model, 1000);\nlogger.info(\"SARSA Bot beats Random Bot in {} % of the games being played\", successRate * 100);\n```\n\nTo save and load the SARSA model:\n\n```java\nString model_json = model.toJson();\nSarsaLearner loaded_from_json = SarsaLearner.fromJson(model_json);\n```\n\nThis sample code can be found in the DojoSarsa.java file in the project.\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchen0040%2Fjava-reinforcement-learning-tic-tac-toe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchen0040%2Fjava-reinforcement-learning-tic-tac-toe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchen0040%2Fjava-reinforcement-learning-tic-tac-toe/lists"}