{"id":27934249,"url":"https://github.com/fracapuano/quinto","last_synced_at":"2026-04-24T16:04:03.439Z","repository":{"id":65633634,"uuid":"581125921","full_name":"fracapuano/Quinto","owner":"fracapuano","description":"Repository for the final project of the \"Computational Intelligence\" course @ PoliTo, 2022/2023","archived":false,"fork":false,"pushed_at":"2023-02-03T12:58:51.000Z","size":21111,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-07T05:43:08.413Z","etag":null,"topics":["board-game","deep-learning","ppo-agent","reinforcement-learning","reinforcement-learning-agent"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fracapuano.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-12-22T10:43:13.000Z","updated_at":"2024-11-08T15:34:08.000Z","dependencies_parsed_at":"2023-02-17T09:15:21.514Z","dependency_job_id":null,"html_url":"https://github.com/fracapuano/Quinto","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/fracapuano/Quinto","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fracapuano%2FQuinto","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fracapuano%2FQuinto/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fracapuano%2FQuinto/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fracapuano%2FQuinto/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fracapuano","download_url":"https://codeload.github.com/fracapuano/Quinto/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fracapuano%2FQuinto/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32230421,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-24T13:21:15.438Z","status":"ssl_error","status_checked_at":"2026-04-24T13:21:15.005Z","response_time":64,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["board-game","deep-learning","ppo-agent","reinforcement-learning","reinforcement-learning-agent"],"created_at":"2025-05-07T05:35:20.010Z","updated_at":"2026-04-24T16:04:03.433Z","avatar_url":"https://github.com/fracapuano.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![alt text](https://i.ibb.co/5jf3wPM/quarto-RL-logo.png)\n# Quinto: Solving Quarto with Reinforcement Learning\nThis repository contains the code necessary to design an end-to-end experimental setting to train a Deep Reinforcement Learning (RL) agent playing the game of Quarto. \n\nThe agent presented in this work is trained leveraging advancements made on On-Policy algorithms and, in particular, the training procedure does employ fairly innovative features such as Action Masking and incremental self-play.\n\n# The game of Quarto\nQuarto is a two-player board game that involves strategy, logic, and luck. The goal of the game is to be the first player to create a line of four pieces, either horizontally, vertically, or diagonally. The game pieces consist of 16 unique pieces with four distinct attributes: shape, height, color, and filling. \nMore formally:\n\n\u003cp align=\"center\"\u003e\n    \u003cimg width=400 src=\"https://i.ibb.co/T05Hc2P/quartopieces.jpg\" alt=\"pieces\"\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n    \u003cem\u003eQuarto Pieces sharing a common property\u003c/em\u003e\n\u003c/p\u003e\n\n\n1. The game of Quarto is played by two players using a 16 piece game set, which consists of 4 by 4 square pieces. Each piece has four characteristics: **height**, **width**, **shape**, and **color**.\n\n2. The purpose of the game is for one player to form a row of four pieces, each with at least one shared characteristic, while the other player tries to prevent it.\n\n\n3. Players alternate turns, with the starting player determined by a coin toss. During each turn, the player must select one of the ones not on the board and offer it to their opponent, who then must place it in a cell of their (the opponent's) choice. The goal of each player is to form a line of four pieces sharing one property, so the piece position is typically chosen according to this aim.\n\n4. If the game reaches a stalemate, with all possible moves blocked, the game is declared a draw.\n\nMore information about the game and how to play can be found [at this page](https://en.wikipedia.org/wiki/Quarto_(board_game)) or at [this YouTube video](https://www.youtube.com/watch?v=v1c-uKD6iOw\u0026themeRefresh=1).\n\n# Installation\nThe project requires Python 3.7 or higher and, among many, the following packages:\n\n- `stable-baselines3`\n- `sb3-contrib`\n- `gym`\n- `torch`\n\nTo reproduce our training routine you must first create a virtual environment in which to install the required dependencies. You can do this via the following: \n```\n$ conda create -n quarto\n$ pip install -r requirements.txt\n```\n\n# Usage\n\n## Reproduce our results\nTo train the agent with default configuration, simply run the following command:\n\n```python\n$ python train.py\n```\n\nThe `train.py` scripts makes usage of several arguments. For the sake of brevity, we do not report each of them here. However, you can read more about them by simply running `python train.py -h`.\n\n### Re-train our best performing agent\n\n**Note:** the default configuration (the one used to train our best-performing agent) produces **`MASKEDPPOv2_100e6.zip`**. This model must later on be fine-tuned with self-play to finally obtain **`MASKEDPPOv3_120e6.zip`**, our best performing agent.\n\nThis procedure can be carried out running the following: \n\n```bash\n$ python train.py  # train MASKEDv2_100e6, takes approximately a week.\n\n$ python train.py --default False --algorithm maskedPPO --train-timesteps 20e6 --test-episodes 100 --losing-penalty True --save-model True --duration-penalty True --action-masking True --model-path commons/trainedmodels/MASKEDv2_100e6.zip --use-symmetries True --self-play True --resume-training True  # insert 20e6 training timesteps when prompted\n```\n\nShould the training process of `MASKEDPPOv2_100e6.zip` crash (as it has been crashing on our machine) you can resume training from one of the checkpoints saved in the `checkpoints` folder, by simply running: \n\n```\n$ python train.py --default False --algorithm maskedPPO --test-episodes 100 --losing-penalty True --save-model True --duration-penalty True --action-masking True --model-path checkpoints/\u003ccheckpointed_model\u003e --resume-training True # insert the number of timesteps you want the model to train after crashing\n```\n\nIn a nutshell, the arguments of `train.py` are used to differentiate the training process for what concerns the **reward function** used and the **opponent** challenged during the training process. \n\nFurthermore, a version of the training process which makes usage of the highly symmetric structure of the game is also available. A precise description of every version implemented so far is presented in the following table. \n\n| **_Environment version_** |            **_Reward Function_**            | **_Uses symmetries_** |       **_Opponent_**       |\n|:-------------------------:|:-------------------------------------------:|:---------------------:|:--------------------------:|\n|            `v0`           |             $r_T = (+1, +0.2, 0)$            |           No          | Valid-Moves-Only Random Opponent |\n|            `v1`           |            $r_T = (+1, +0.2, -1)$            |           No          | Valid-Moves-Only Random Opponent |\n|            `v2`           | $r_T = (5 - 0.75 \\cdot (T/2 - 2), +0.5, -1)$ |           No          | Valid-Moves-Only Random Opponent |\n|            `v3`           | $r_T = (5 - 0.75 \\cdot (T/2 - 2), +0.5, -1)$ |  Yes, rotational only |          Self-play         |\n\nWhere $r_T$ indicates the reward obtained by the agent in the terminal state (namely, a state in which it wins, loses or draws the game). The elements of $r_T$ represent the reward obtained for winning, drawing and losing (respectively) a training game. \n\nIt is worth mentioning that $r_t = 0 \\ \\forall t \\neq T$ ( $r$ is, thefore, a *sparse* reward). \n\nSince we wanted our agent to be as general as possible, we avoided providing intermediate rewards for reaching certain board configurations. \nThis was a clear design choice that, while clearly made the training process way more complex, prevented us to bias the agent towards a particular (not necessarily optimal) behavior.\n\nDuring the training process, the agent would be periodically saved in a `checkpoint` folder (that would also be used to load periodically dump and load self-playing agents). \n\nOnce a model is fully trained, it is saved in the `commons/trainedmodels` folder. During the training process the policy trained is optionally tested againts its own opponent for an arbitrary number of games, in the sake of more accurately diagnosing the characteristics of the training process. These data files are stored in the `logs` file.\n\nModels are saved in the `\u003cALGORITHM\u003e\u003cVERSION\u003e_\u003cTRAININGTIMESTEPS\u003e.zip` format, whereas logs are saved in the `\u003cALGORITHM\u003e\u003cVERSION\u003e_\u003cTRAININGTIMESTEPS\u003e_logfile.txt`.\n\nFor the sake of completeness, we also report here the time needed to train these models.\n\n\n| **_Algorithm_** | **_Version_** | **_Timesteps_** | **_Training time_** |\n|:---------------:|:-------------:|:---------------:|:-------------------:|\n|     **PPO**     |       v0      |       5e6       |       _6h 29m_      |\n|     **PPO**     |       v1      |       5e6       |       _4h 34m_      |\n|     **A2C**     |       v0      |       5e6       |       _5h 43m_      |\n|     **A2C**     |       v1      |       5e6       |       _4h 52m_      |\n|  **MaskedPPO**  |       v0      |       5e6       |       _6h 27m_      |\n|  **MaskedPPO**  |       v1      |       5e6       |       _7h 28m_      |\n|  **MaskedPPO**  |       v2      |      100e6      |       _~1 week_      |\n|  **MaskedPPO**  |       v3      |   (100 + 20)e6  |   _~1 week + 27 hours_  |\n\n\nPlease note that the last model presented here is nothing but an instance of the the `MASKEDPPOv2_100e6` model incrementally trained with self-play and symmetries for an additional 20M timesteps.\n\nMore details on the training procedures and the rationale behind the algorithm choices can be found in our report.\n\nOur best performing model can be found at: `commons/trainedmodels/MASKEDPPOv3_120.zip`. \n\n## Loading a trained model\n\nDepending on the model used, it is necessary to use different objects (either from `stable-baselines3` or `sb3-contrib`) to correctly load the state dict characteristic of each model.\n\nTo load a A2C or PPO-based model simply run (as per the official documentation): \n\n```python\nfrom stable_baselines3 import A2C, PPO\nfrom sb3_contrib import MaskablePPO\nfrom commons.quarto.objects import Quarto\nfrom main import RLPlayer\n\n\nA2C_model = A2C.load(\"commons/trainedmodels/A2Cv1_5e6.zip\")\nPPO_model = PPO.load(\"commons/trainedmodels/PPOv0_5e6.zip\")\nmaskedPPO_model = MaskablePPO.load(\n            'commons/trainedmodels/MASKEDPPOv3_130e6.zip',\n            custom_objects= {\n                \"learning_rate\": 0.0,\n                \"lr_schedule\": lambda _: 0.0, \n                \"clip_range\": lambda _: 0.0\n            }\n)\n# create instance of the game \ngame = Quarto()\n# create player\nplayerRL = RLPlayer(game, maskedPPO_model)  # can also be A2C_model or PPO_model\n```\n\nThe fact that models are wrapped by the `Player` class (defined in `commons/quarto/objects`) is justified by the fact that we wanted our players to be compatible with the common interface presented [here](https://github.com/squillero/computational-intelligence/tree/master/2022-23/quarto) and reproduced in this repo in the `main.py` file.\n\nOur players are modelled as instances of the `RLPlayer(...)`.\n\n## Experiments \u0026 Results\nAn extensive discussion of our experimental results can be found in the [full report](https://github.com/fracapuano/Quinto/blob/main/Quinto_Report.pdf) that comes with this repo. \n\nOur experiments related to the training phase of `v3` can be found [here](https://wandb.ai/francescocapuano/QuartoRL-v3%20training?workspace=user-). \n\nCurrently, we are still experimenting with a last version fully trained with self-play for a larger number of episodes. \nThese experiments can be consulted [here](https://wandb.ai/francescocapuano/QuartoRL-v2%20seedless%20training?workspace=user-). \n\nTested against a randomly playing agent, our algorithm peaks 90%+ winning rate over 100 games. More interestingly, however, our approach allows running **more than 50 games a second**.\n\nIn particular, our algorithm can play 1000 games against a random agent, win the vast majority of these matches while requiring 50 seconds only. \n\n\n\n## Credits\nWhile we take pride in the full pathernity of this work, we would like to acknowledge the influence of several online open-source resources such as: \n\n1. [This repo](https://github.com/benallard/quarto-gym) has been used as an initial skeleton to develop this project. While we significantly expanded its code-base, having it allowed us to start on the right foot.\n\n2. [This piece of documentation](https://sb3-contrib.readthedocs.io/en/master/modules/ppo_mask.html) helped us developing the action-masking framework we have been using in this report.\n\n3. [This repo](https://github.com/hardmaru/slimevolleygym/blob/master/TRAINING.md) played a major role in the implementation of our self-play paradigm.\n\n4. As usual, countless visits to [Stack Overflow](https://stackoverflow.com/) have been a key part of the development process.\n\n5. [This paper](https://www.researchgate.net/publication/261848662_An_artificial_intelligence_for_the_board_game_'Quarto'_in_Java) has helped us in better framing the problem. \n\n6. Lastly, [this book](https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf) also played a huge role in our RL-based approach to this problem.\n\n7. Lately, we have been experimenting with [ChatGPT](https://chat.openai.com/) for brainstorming and implementation tweaks.\n\n# Authors\nFrancesco Capuano ([@fracapuano](https://github.com/fracapuano)), Matteo Matteotti ([@mttmtt31](https://github.com/mttmtt31))\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffracapuano%2Fquinto","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffracapuano%2Fquinto","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffracapuano%2Fquinto/lists"}