{"id":16598516,"url":"https://github.com/guidopetri/chess-pipeline","last_synced_at":"2025-07-25T14:03:18.680Z","repository":{"id":110337405,"uuid":"188560390","full_name":"guidopetri/chess-pipeline","owner":"guidopetri","description":"Pulling games from the Lichess API into a PostgreSQL database for data analysis.","archived":false,"fork":false,"pushed_at":"2025-01-21T04:01:18.000Z","size":3479,"stargazers_count":19,"open_issues_count":26,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-27T14:31:14.043Z","etag":null,"topics":["chess","luigi-pipeline","postgresql","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/guidopetri.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-05-25T11:54:08.000Z","updated_at":"2025-02-03T18:07:46.000Z","dependencies_parsed_at":"2024-04-06T00:30:34.929Z","dependency_job_id":"3df47b73-1fc3-4a19-9a85-eeaa382f5070","html_url":"https://github.com/guidopetri/chess-pipeline","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guidopetri%2Fchess-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guidopetri%2Fchess-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guidopetri%2Fchess-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guidopetri%2Fchess-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/guidopetri","download_url":"https://codeload.github.com/guidopetri/chess-pipeline/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243830939,"owners_count":20354853,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chess","luigi-pipeline","postgresql","python"],"created_at":"2024-10-12T00:08:51.918Z","updated_at":"2025-03-16T21:30:48.161Z","avatar_url":"https://github.com/guidopetri.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Chess games pipeline\n\nThis is a simple scripting package to pull games from [Lichess](http://lichess.org) into a PostGreSQL database.\n\n## Requirements\n\nIn order to run it, the following Python packages are required:\n\n- luigi\n- psycopg2\n- pandas\n- python-chess\n- python-lichess\n\nA PostGreSQL server must also be set up properly. The data is written to a table with the format listed in `chess_games.sql`.\n\n## Gathering chess data\n\nTo gather data from Lichess, run the following command with the location of `chess_pipeline.py` in your PYTHONPATH:\n\n`luigi --module chess_pipeline CopyGames`\n\nBy default, the script pulls Blitz games from the last two days for the user [`thibault`](http://lichess.org/@/thibault). This can be changed by passing in the following arguments:\n\n- `--player`, for the chess player\n- `--perf-type`, for the kind of chess game - bullet, blitz, classical, etc.\n- `--since`, for since when to pull. This is given in Unix time.\n- `--single-day`, if only a single day is to be pulled from the API (this is a boolean flag)\n\nThe above can also be added to your `crontab`, as long as your PATH and PYTHONPATH are set up correctly.\n\nThe script defaults to writing to a (main) table called `chess_games`. It also writes to tables called `game_clocks`, `game_evals`, and `game_moves` for the clock information, the computer evaluation of the position, and the moves made in the game, respectively.\n\nIn order to access these tables, a section named `postgres_cfg` in your `luigi.cfg` file containing the following keys must exist:\n\n- `user`, the PostGreSQL user to log in as\n- `password`, the password for the above user\n- `host`, the hostname for where the server is running\n- `port`, the port to which the server is listening\n- `database`, the database to write to\n\nFor greater control over write permissions, the following keys are used whenever data is read from the PostGreSQL server:\n\n- `read_user`, the PostGreSQL (read-only) user to log in as\n- `read_password`, the password for the above user\n\nOptionally, you can also add a `lichess_token` section with the following key:\n\n- `lichess-token` is the lichess API token to be used for faster API calls\n\n### Local Stockfish\n\nSome games don't have server-side analyses available for them. To allow analysis of these games, a local stockfish can be used to analyze each position and create an evaluation for it. This requires the `stockfish` library for conversing with stockfish. Using local analysis is as simple as adding the flag:\n\n- `--local-stockfish`\n\nAdditionally, you also need a `stockfish_cfg` section with the keys:\n\n- `depth`, the depth at which stockfish will analyze\n- `location`, the location of the stockfish executable\n\nDepending on the processing power of your machine, you might want to choose a low depth - analyzing all the positions takes a while. Server-side analyses are depth 20.\n\n## Attributes\n\nFor each chess game:\n  - game info (variant)\n  - game result\n  - game link\n  - how the game finished (time forfeit/resignation)  \n  - player name\n  - player color\n  - rating diff for player\n  - player result\n  - game type (time control - bullet, blitz, etc.)\n  - datetime played\n  - clock start time\n  - clock increment\n  - in arena or not\n  - rated or casual\n  - player rating\n  - queen exchange or not\n  - player castling side\n  - opening name\n  - opening ECO\n  - move list\n  - move evaluations (server-side if available, otherwise with stockfish if enabled)\n  - clock times per move\n\nand similar columns for opponent.\n\n## Newsletter\n\nIt's also possible to send out a newsletter with some simple graphs/metrics on how your chess games have been for the week. However, there are a few additional Python package requirements for this:\n\n- seaborn\n- beautifulSoup4\n\nAdditionally, two config sections need to be added:\n\n- `newsletter_cfg`, with a sender email under the key `sender`\n- `sendgrid`, with an API key under the key `apikey`\n\nThe newsletter can be sent with the following command:\n\n`luigi --module newsletter SendNewsletter`\n\nwith the options:\n\n- `--player`, for the player whose games we're going to analyze\n- `--receiver`, for the email address to send to\n\nCurrently, the newsletter sends out win rates (in %) for black/white in each time control category, as well as a rating progression chart by day, along with min/maxes for each day.\n\n## Running the test suite\n\nThe tests are written using `pytest`. To run the test suite, run the following from the root directory:\n\n```bash\nmake pytest\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fguidopetri%2Fchess-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fguidopetri%2Fchess-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fguidopetri%2Fchess-pipeline/lists"}