https://github.com/woctezuma/metacouncil-goty
MetaCouncil's PC Games of the Year Awards 2018-20, Decade Awards 201X ; SteamDB's Discord GotY Awards 2024.
https://github.com/woctezuma/metacouncil-goty
difflib goty igdb igdb-api igdb-database levenshtein-distance metacouncil schulze schulze-method schulze-voting steamcouncil steamspy steamspy-api vote
Last synced: 8 months ago
JSON representation
MetaCouncil's PC Games of the Year Awards 2018-20, Decade Awards 201X ; SteamDB's Discord GotY Awards 2024.
- Host: GitHub
- URL: https://github.com/woctezuma/metacouncil-goty
- Owner: woctezuma
- License: mit
- Created: 2019-01-02T12:30:02.000Z (almost 7 years ago)
- Default Branch: master
- Last Pushed: 2025-03-18T18:58:38.000Z (8 months ago)
- Last Synced: 2025-03-18T19:51:58.414Z (8 months ago)
- Topics: difflib, goty, igdb, igdb-api, igdb-database, levenshtein-distance, metacouncil, schulze, schulze-method, schulze-voting, steamcouncil, steamspy, steamspy-api, vote
- Language: Python
- Homepage:
- Size: 1010 KB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# MetaCouncil's PC Games of the Year (GOTY) Awards
[![Build status with Github Action][build-image-action]][build-action]
[![Code coverage][codecov-image]][codecov]
[![Code Quality][codacy-image]][codacy]
![MetaCouncil GOTY banner][generic-goty-cover]
## Introduction
These awards focus on PC games which we deem GOTY worthy, as well as the best DLC / Ongoing game, the best Early Access
game, and the best VR game. Exceptionally, for 2019, there is also an award for the Game of the Decade (GOTD).
## Method
Ballots are cast on MetaCouncil ([2018][ballots-2018], [2019][ballots-2019], [2020][ballots-2020])
and processed with [our implementation](https://github.com/woctezuma/steam-era-goty) of Schulze method.
## Usage
- To compute the GOTY ranking, run:
```bash
python schulze_goty.py
```
- To compute the GOTD ranking, run:
```bash
python schulze_gotd.py
```
- To compute the rankings for the optional categories (DLC / Ongoing, Early Access, VR, etc.), run:
```bash
# if there is a single choice per optional category:
python optional_categories.py
# if there are several (ranked) choices per optional category:
python schulze_optional.py
```
- To generate a message to thank voters, run:
```bash
python generate_thanks.py
```
- To anonymize data before uploading it to Github, run:
```bash
python anonymize_data.py
```
### With SteamSpy
- If needed, edit hard-coded values below, then run the three aforementioned scripts again:
- `extend_steamspy.py` (manual addition of a few appIDs to SteamSpy's database)
- `hard_coded_matches.py` (manual match of a few game names with appIDs)
- `disqualify_vote.py` (manual disqualification of a few appIDs)
- `whitelist_vote.py` (manual white-listing of a few appIDs)
### With IGDB
Let us assume the target release year is `2018` for this section, i.e. we focus on games released during the year 2018.
- If needed, edit hard-coded values below, then run the three aforementioned scripts again:
- `fixes_to_igdb_local_database_2018.json` (manual addition of a few appIDs to IGDB's database)
- `fixes_to_igdb_match_database_2018.json` (manual match of a few game names with appIDs)
- `disqualified_igdb_ids_2018.json` (manual disqualification of a few appIDs)
- `whitelisted_igdb_ids_2018.json` (manual white-listing of a few appIDs)
The fixes to the local database allow to extend the IGDB database, in case a game is missing from IGDB, e.g. Steam games in the adult section of the store.
The fixes to the match database allow to manually enforce matches between input game names and IGDB ids, in order to
edit results from the automatic matching method:
- fix empty matches: no match could be found by IGDB,
- fix actual mismatches: the match is factually wrong,
- merge matches for different versions (vanilla, definitive, etc.) of the same game, e.g. Darks Souls 1.
Without merging versions of the same game, votes for this same game would be spread to the detriment of the game rank.
The black-list allows to disqualify some games for manually specified reasons.
The white-list allows to prevent the automatic disqualification of some games due to their reported release date differing from the target release year.
## Results
Results are displayed:
- on the Wiki:
- [GotY 2018][results-2018-wiki],
- [GotY 2019][results-2019-wiki],
- [GotD 201X][results-201X-wiki],
- [GotY 2020][results-2020-wiki],
- [GotY 2024][results-2024-wiki],
- on MetaCouncil:
- [GotY 2018][results-2018],
- [GotY 2019][results-2019],
- [GotD 201X][results-201X],
- [GotY 2020][results-2020].
## Alternative methods for game name matching
### SteamSpy and Levenshtein distance
The first implementation relied on [SteamSpy](https://github.com/woctezuma/steamspypi)'s database, and matched game names with [Levenshtein distance](https://github.com/ztane/python-Levenshtein).
In practice, Levenshtein distance is effective at fixing typos, but cannot work for cases where the input is short and the actual game title is long, e.g. for [Resident Evil 7](https://store.steampowered.com/app/418370/):
> RESIDENT EVIL 7 biohazard / BIOHAZARD 7 resident evil
### SteamSpy and difflib
If we use SteamSpy, we have the choice between Levenshtein distance and difflib for name matching.
[Difflib](https://docs.python.org/3/library/difflib.html) allows to match game names with the longest contiguous matching subsequence.
However, difflib is notably slower than Levenshtein distance.
### IGDB
As of January 2020, [IGDB](https://www.igdb.com/api)'s database, which extends beyond Steam, can be used in place of SteamSpy with the `use_igdb` flag.
If IGDB is to be used, then you need to have a user secret key to be authorized.
It is free, but you need to register on IGDB.
The API is queried whenever it is necessary, and the responses are locally saved to avoid unnecessary requests.
As a free user, there is a monthly allowance of 50k requests per month.
Name matching is delegated to IGDB because the whole IGDB database is not locally available.
In theory, this could lead to worse results if there are typos in the input names.
However:
- IGDB's database is larger than SteamSpy's, so name matching could be better, thanks to the availability of alternative names.
- IGDB also offers access to information about DLC, which could be useful for at least one of the optional categories,
- typos are not a big issue: they are rare in the input game names for the GotY votes.
### Benchmark
A quantitative comparison is shown [in a benchmark](https://github.com/woctezuma/metacouncil-goty/wiki/Benchmark) on the Wiki.
The mismatches observed with the 2018 dataset are counted, and the best performing methods are:
1. IGDB database with a constraint w.r.t. the release year: **8 mismatches**,
2. IGDB database: 11 mismatches,
3. vanilla SteamSpy database with difflib matching and a constraint w.r.t. the release year: 12 mismatches,
4. vanilla SteamSpy database with difflib matching: 14 mismatches,
5. vanilla SteamSpy database with Levenshtein distance: 18 mismatches (same performance with and without constraint).
In summary, in order to minimize the number of manual edits necessary to extend the database and to fix name mismatches,
the most promising method involves using the IGDB database with a constraint w.r.t. the release year.
## References
- [Ranked-choice voting](https://en.wikipedia.org/wiki/Ranked_voting)
- [Schulze method](https://en.wikipedia.org/wiki/Schulze_method)
- [SteamSpy API](https://github.com/woctezuma/steamspypi)
- [IGDB API](https://www.igdb.com/api)
- [Levenshtein distance](https://github.com/ztane/python-Levenshtein)
- [difflib](https://docs.python.org/3/library/difflib.html): the longest contiguous matching subsequence
[build]:
[build-image]:
[build-action]:
[build-image-action]:
[pyup]:
[dependency-image]:
[python3-image]:
[codecov]:
[codecov-image]:
[codacy]:
[codacy-image]:
[generic-goty-cover]:
[ballots-2018]:
[ballots-2019]:
[ballots-2020]:
[results-2018]:
[results-2019]:
[results-201X]:
[results-2020]:
[results-2018-wiki]:
[results-2019-wiki]:
[results-201X-wiki]:
[results-2020-wiki]:
[results-2024-wiki]: