{"id":20804939,"url":"https://github.com/lichess-org/lila-openingexplorer","last_synced_at":"2025-04-04T18:05:20.074Z","repository":{"id":37447273,"uuid":"50526057","full_name":"lichess-org/lila-openingexplorer","owner":"lichess-org","description":"Opening explorer for lichess.org that can handle all the variants and trillions of unique positions","archived":false,"fork":false,"pushed_at":"2025-03-01T23:29:08.000Z","size":2134,"stargazers_count":137,"open_issues_count":7,"forks_count":36,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-03-28T17:04:45.283Z","etag":null,"topics":["chess","chess-database","http-api","lichess"],"latest_commit_sha":null,"homepage":"http://lichess.org/analysis#explorer","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lichess-org.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"custom":"https://lichess.org/patron"}},"created_at":"2016-01-27T17:48:36.000Z","updated_at":"2025-02-19T02:02:39.000Z","dependencies_parsed_at":"2023-02-18T22:46:05.899Z","dependency_job_id":"27c902c2-9bbb-4dc3-9b05-1f032cb728a9","html_url":"https://github.com/lichess-org/lila-openingexplorer","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lichess-org%2Flila-openingexplorer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lichess-org%2Flila-openingexplorer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lichess-org%2Flila-openingexplorer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lichess-org%2Flila-openingexplorer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lichess-org","download_url":"https://codeload.github.com/lichess-org/lila-openingexplorer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247226213,"owners_count":20904465,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chess","chess-database","http-api","lichess"],"created_at":"2024-11-17T19:12:29.683Z","updated_at":"2025-04-04T18:05:20.047Z","avatar_url":"https://github.com/lichess-org.png","language":"Rust","funding_links":["https://lichess.org/patron"],"categories":[],"sub_categories":[],"readme":"lila-openingexplorer\n====================\n\n[![Test](https://github.com/lichess-org/lila-openingexplorer/actions/workflows/test.yml/badge.svg)](https://github.com/lichess-org/lila-openingexplorer/actions/workflows/test.yml)\n\n[Opening explorer](https://lichess.org/blog/Vs0xMTAAAD4We4Ey/opening-explorer)\nfor lichess.org, capable of handling trillions of positions, featuring:\n\n* A database of master games\n* [Rated games from Lichess itself](https://database.lichess.org/)\n* An on-demand database of [openings by player](https://lichess.org/blog/YXMPxxMAACEAy3g4/announcing-the-personal-opening-explorer)\n* [Curated opening names](https://github.com/lichess-org/chess-openings)\n\nUsage\n-----\n\n### Run server\n\n1. Install recent stable Rust ([rustup](https://rustup.rs/) recommended).\n\n2. Set some environment variables used at build time: `set -a \u0026\u0026 source .env \u0026\u0026 set +a`\n\n3. Build the server and view available options:\n\n   ```\n   cargo run --release -- --help\n   ```\n\n   Strongly consider adjusting `--db-compaction-readahead`, `--db-cache`, and\n   `--db-rate-limit` depending on your setup.\n\n4. Run the server with the chosen options:\n\n   ```\n   ulimit -n 131072 \u0026\u0026 EXPLORER_LOG=lila_openingexplorer=info cargo run --release -- --db-compaction-readahead\n   ```\n\n:warning: In a production environment, administrative endpoints must be\nprotected using a reverse proxy.\nIt's best to whitelist only `/masters`, `/lichess`, and `/player`.\n\n### Import games\n\n1. Download database dumps from https://database.lichess.org/.\n\n2. Import (optionally works directly with compressed files):\n\n   ```\n   cd import-pgn\n   cargo run --release -- *.pgn.zst\n   ```\n\n   The database size will be well below 3x the compressed PGN size.\n\n   If you can fit this on SSDs, read and compaction performance, especially\n   tail latencies, will benefit significantly. All else equal, RAIDs with\n   multiple small disks are preferable to RAIDs with few larger disks.\n   Block and page cache will take advantage of large amounts of available RAM.\n\n   As of February 2023, Lichess handles around 12k requests/minute, using:\n\n   * 4 spinning disks in RAID10\n   * 128 GiB RAM, of which 100 GiB are used for the block cache\n   * Compressed PGN import rate of around 100 KiB/s on the live system,\n     paused at peak hours.\n\n   Importing is currently very slow, but good enough to index faster\n   than games are played. Initially, the database was prepared\n   offline, with speed dropping as the database grew, averaging 1 MiB/s\n   compressed indexing speed (so effectively 7 MiB/s uncompressed PGN data).\n\nMonitoring\n----------\n\n### `/monitor`\n\nExample:\n\n```\ncurl http://localhost:9002/monitor\n```\n\nResponse in InfluxDB line protocol:\n\n```\nopening_explorer block_index_miss=2271815u,block_index_hit=44204637u,block_filter_miss=2272244u,block_filter_hit=81741291u,block_data_miss=31540587u,block_data_hit=33327789u,indexing=5u,lichess_cache=31038u,lichess_miss=2993390u,lichess_history_cache=2112u,lichess_history_miss=19558u,masters_cache=38276u,masters_miss=3430066u,masters=158629555u,masters_game=2519908u,lichess=121970833029u,lichess_game=4331746117u,player=18693470276u,player_status=182129u\n```\n\n### `/monitor/db/\u003cprop\u003e`\n\n### `/monitor/cf/\u003ccf\u003e/\u003cprop\u003e`\n\nExample:\n\n```\ncurl http://localhost:9002/monitor/cf/lichess/rocksdb.stats\n```\n\n```\n** Compaction Stats [lichess] **\nLevel    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)\n------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n  L0      1/0   34.75 MB   0.2     64.3     0.0     64.3     157.5     93.2       0.0   1.6      1.4      3.5  46068.07           3002.68      2760   16.691   1763M    92M       0.0       0.0\n  L1      4/0   205.48 MB   0.8    172.6    93.2     79.4     165.5     86.1       0.0   1.8      3.7      3.5  48254.63           3779.43       207  233.114   4758M   164M       0.0       0.0\n  L2     52/0    2.49 GB   1.0    321.9    81.7    240.3     317.2     77.0       4.4   3.9      3.5      3.4  94232.38           6840.29      1345   70.061   8967M   121M       0.0       0.0\n  L3    591/0   24.95 GB   1.0    338.4    78.2    260.1     333.0     72.9       3.1   4.3      3.5      3.5  97870.15           7213.25      1532   63.884   9448M   146M       0.0       0.0\n  L4   3367/0   90.71 GB   0.4     81.3    48.6     32.7      79.9     47.2      27.5   1.6      3.7      3.6  22451.14           1770.16       904   24.835   2281M    35M       0.0       0.0\n  L6  47776/0    2.78 TB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0       0.0       0.0\n Sum  51791/0    2.90 TB   0.0    978.5   301.6    676.9    1053.2    376.3      35.0  10.9      3.2      3.5 308876.38          22605.81      6748   45.773     27G   560M       0.0       0.0\n Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0       0.0       0.0\n\n** Compaction Stats [lichess] **\nPriority    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)\n---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n Low      0/0    0.00 KB   0.0    978.5   301.6    676.9     956.6    279.7       0.0   0.0      3.5      3.4 288675.78          20752.87      4203   68.683     27G   560M       0.0       0.0\nHigh      0/0    0.00 KB   0.0      0.0     0.0      0.0      96.6     96.6       0.0   0.0      0.0      4.9  20200.60           1852.94      2545    7.937       0      0       0.0       0.0\n\nBlob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0\n\nUptime(secs): 610745.6 total, 3.5 interval\nFlush(GB): cumulative 96.595, interval 0.000\nAddFile(GB): cumulative 0.000, interval 0.000\nAddFile(Total Files): cumulative 0, interval 0\nAddFile(L0 Files): cumulative 0, interval 0\nAddFile(Keys): cumulative 0, interval 0\nCumulative compaction: 1053.16 GB write, 1.77 MB/s write, 978.51 GB read, 1.64 MB/s read, 308876.4 seconds\nInterval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds\nStalls(count): 1028 level0_slowdown, 1018 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 1809 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count\nBlock cache LRUCache@0x7feb60a3f150#4001884 capacity: 40.00 GB usage: 39.97 GB table_size: 524288 occupancy: 6030149 collections: 1018 last_copies: 5 last_secs: 0.039008 secs_since: 408\nBlock cache entry stats(count,size,portion): DataBlock(165238,10.35 GB,25.885%) FilterBlock(22233,29.11 GB,72.7832%) IndexBlock(22348,499.64 MB,1.21981%) Misc(1,0.00 KB,0%)\n\n** File Read Latency Histogram By Level [lichess] **\n\n** DB Stats **\nUptime(secs): 610745.6 total, 3.5 interval\nCumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s\nCumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s\nCumulative stall: 00:00:0.000 H:M:S, 0.0 percent\nInterval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s\nInterval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s\nInterval stall: 00:00:0.000 H:M:S, 0.0 percent\n```\n\nPublic HTTP API\n---------------\n\nSee https://lichess.org/api#tag/Opening-Explorer.\n\n### `/masters`\n\n### `/lichess`\n\n### `/player`\n\nExample:\n\n```\ncurl https://explorer.lichess.ovh/player?player=foo\u0026color=white\u0026play=e2e4\n```\n\nQuery parameters:\n\nname | type | default | description\n--- | --- | --- | ---\nvariant | string | `chess` | `antichess`, `atomic`, `chess` (or `standard`, `chess960`, `fromPosition`), `crazyhouse`, `horde`, `kingOfTheHill`, `racingKings`, `threeCheck`\nfen | string | *starting position of variant* | FEN of the root position\nplay | string | *empty* | Comma separated moves in UCI notation. Play additional moves starting from *fen*. Required to find an opening name, if *fen* is not an exact match for a named position.\nplayer | string | *required* | Username to filter for\ncolor | string | *required* | Filter for games where *player* is `white` or `black`\nmodes | string | *all* | Comma separated list of game modes (`rated`, `casual`) to filter for\nspeeds | string | *all* | Comma separated list of speeds (`ultraBullet`, `bullet`, `blitz`, `rapid`, `classical`, `correspondence`) to filter for\nsince | string | `0000-01` | Year-Month. Filter for games played in this month or later\nuntil | string | `3000-12` | Year-Month. Filter for games played in this month or earlier\n\nResponse: Streamed [`application/x-ndjson`](https://github.com/ndjson/ndjson-spec)\nwith rows as follows.\n\nWill start indexing, immediately respond with the current results,\nand stream more updates until indexing is complete. The stream is throttled\nand deduplicated. Empty lines may be sent to avoid timeouts.\n\n```js\n{\n    \"white\": 10, // total number of white wins from this position\n    \"draws\": 1,\n    \"black\": 22,\n    \"moves\": [\n        {\n            \"uci\": \"e7e5\",\n            \"san\": \"e5\",\n            \"white\": 6, // total number of white wins with this move.\n                        // more may transpose to resulting position.\n            \"draws\": 1,\n            \"black\": 9,\n            \"averageOpponentRating\": 1500, // or null\n            \"game\": { // only game for this move.\n                      // would not actually be sent, because there are multiple\n                      // games in this case, but for example:\n                \"id\": \"uPdCG6Ts\",\n                \"winner\": \"black\",\n                \"speed\": \"correspondence\",\n                \"mode\": \"casual\",\n                \"white\": {\n                    \"name\": \"foo\",\n                    \"rating\": 1500\n                },\n                \"black\": {\n                    \"name\": null,\n                    \"rating\": null\n                },\n                \"year\": 2015,\n                \"month\": \"2015-09\"\n            }\n        },\n        // ...\n    ],\n    \"recentGames\": [ // currently up to 15 recent games.\n                     // limit is up to discussion.\n        {\n            \"uci\": \"e7e5\",\n            \"id\": \"uPdCG6Ts\",\n            \"winner\": \"black\",\n            \"speed\": \"correspondence\",\n            \"mode\": \"casual\",\n            \"white\": {\n                \"name\": \"foo\",\n                \"rating\": 1500\n            },\n            \"black\": {\n                \"name\": null,\n                \"rating\": null\n            },\n            \"year\": 2015,\n            \"month\": \"2015-09\"\n        },\n        // ...\n    ],\n    \"opening\": {\n        \"eco\": \"B00\",\n        \"name\": \"King's Pawn\"\n    },\n    \"queuePosition\": 3 // waiting for other players to be indexed first\n}\n```\n\nLicense\n-------\n\nLicensed under the GNU Affero General Public License v3. See the `LICENSE` file\nfor details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flichess-org%2Flila-openingexplorer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flichess-org%2Flila-openingexplorer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flichess-org%2Flila-openingexplorer/lists"}