{"id":50634655,"url":"https://github.com/getjump/protondb-game-compatibility-prediction","last_synced_at":"2026-06-07T01:01:54.040Z","repository":{"id":344538479,"uuid":"1182144908","full_name":"getjump/protondb-game-compatibility-prediction","owner":"getjump","description":"ML-based game compatibility prediction for Proton/Steam on Linux","archived":false,"fork":false,"pushed_at":"2026-03-15T05:23:27.000Z","size":576,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-15T16:49:46.162Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/getjump.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-15T05:11:31.000Z","updated_at":"2026-03-15T05:23:30.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/getjump/protondb-game-compatibility-prediction","commit_stats":null,"previous_names":["getjump/protondb-game-compatibility-prediction"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/getjump/protondb-game-compatibility-prediction","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getjump%2Fprotondb-game-compatibility-prediction","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getjump%2Fprotondb-game-compatibility-prediction/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getjump%2Fprotondb-game-compatibility-prediction/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getjump%2Fprotondb-game-compatibility-prediction/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/getjump","download_url":"https://codeload.github.com/getjump/protondb-game-compatibility-prediction/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getjump%2Fprotondb-game-compatibility-prediction/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34005030,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-06T02:00:07.033Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-07T01:01:53.041Z","updated_at":"2026-06-07T01:01:54.020Z","avatar_url":"https://github.com/getjump.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ProtonDB Game Compatibility Prediction\n\nML pipeline that predicts game compatibility with Proton/Wine on Linux using [ProtonDB](https://www.protondb.com/) community reports.\n\n**Two-stage cascade LightGBM classifier** with IRT-based label denoising, trained on 350K+ user reports with 123 features.\n\n### Results\n\n| Evaluation | 3-class F1 | Binary F1 | Accuracy |\n|---|---|---|---|\n| Per-report | 0.780 | 0.906 | 0.828 |\n| **Per-game (production)** | **0.871** | **0.943** | **0.934** |\n\nPer-game predictions aggregate individual report predictions via majority vote — individual errors cancel out.\n\n## Architecture\n\n```\nProtonDB dump → Worker → SQLite DB → Preprocessing → ML Training → Prediction\n                                          │\n                          ┌────────────────┼────────────────┐\n                          │                │                │\n                    Enrichment      Normalization     LLM extraction\n                  (Steam, PCGW,    (GPU/CPU/driver    (launch options,\n                   ProtonDB API,    heuristic)         text analysis)\n                   Steam PICS,\n                   AWACY, GitHub)\n```\n\n**Stack:** Python 3.12, LightGBM, FastAPI, SQLite, SHAP, Click CLI.\n\n### Key components\n\n- **Worker** fetches ProtonDB data dumps and imports reports into SQLite\n- **Preprocessing** enriches data from Steam Store, Steam PICS (bulk CM protocol), PCGamingWiki, AreWeAntiCheatYet, ProtonDB API (contributor data); normalizes GPU/CPU strings; extracts structured data from text via LLM\n- **ML** trains a cascade classifier with IRT denoising, SVD embeddings, and per-game aggregation\n- **IRT (Item Response Theory)** decomposes subjective tinkering/oob labels into per-annotator strictness and per-game difficulty — the key innovation (+0.030 F1)\n\n### Key innovations\n\n1. **IRT label denoising** — 1PL Item Response Theory separates annotator bias (θ) from game difficulty (d). Resolves the subjective tinkering↔works_oob boundary that causes 15-20% label noise. (+0.030 F1)\n2. **Contributor-aware relabeling** — replaces Cleanlab + rule-based heuristics with IRT-informed relabeling based on annotator strictness. (+0.017 F1)\n3. **Per-game majority vote** — aggregates per-report predictions at inference time. Errors cancel out, boosting F1 from 0.780 to 0.871. (+0.091 F1)\n\n## Setup\n\n```bash\ngit clone https://github.com/getjump/protondb-game-compatibility-prediction\ncd protondb-game-compatibility-prediction\n\npython -m venv .venv\nsource .venv/bin/activate\npip install -e \".[dev]\"\n\ncp .env.example .env\n# Edit .env — configure LLM backend (local ollama, OpenRouter, or Claude CLI)\n```\n\n## Usage\n\n### 1. Import data\n\n```bash\n# Download and import the latest ProtonDB dump\nprotondb-settings worker check\nprotondb-settings worker sync\n```\n\n### 2. Preprocess\n\n```bash\n# Cleaning + heuristic normalization + enrichment (no LLM required)\nprotondb-settings preprocess run\n\n# Enrichment from additional sources\nprotondb-settings preprocess run --step enrichment --source protondb_reports\nprotondb-settings preprocess run --step enrichment --source steam_pics\n\n# LLM-based preprocessing (requires local or cloud LLM)\nprotondb-settings preprocess llm normalize-gpu\nprotondb-settings preprocess llm normalize-cpu\nprotondb-settings preprocess llm parse-launch-options\nprotondb-settings preprocess llm extract\n\n# LLM backends: --backend openai (default), openrouter, or claude-cli\nprotondb-settings preprocess llm --backend openrouter extract\n```\n\n### 3. Train\n\n```bash\nprotondb-settings ml train-cascade\n# Reuse Stage 1 for faster iteration on Stage 2:\nprotondb-settings ml train-cascade --reuse-stage1 data/model_stage1.pkl\n```\n\n### 4. Serve (WIP)\n\n```bash\nprotondb-settings serve\n```\n\n## How it works\n\n### Model\n\nTwo-stage cascade with IRT denoising:\n\n1. **Stage 1:** borked (0) vs works (1) — catches broken games (F1=0.846)\n2. **Stage 2:** tinkering (0) vs works_oob (1) — for non-broken games, distinguishes \"needs tweaking\" from \"works out of the box\" (F1=0.880 tinkering, 0.614 oob)\n3. **IRT fitting:** decomposes annotator strictness (θ) and game difficulty (d) from contributor data\n4. **Contributor-aware relabeling:** corrects labels from strict annotators\n5. **Per-game aggregation:** majority vote at inference time\n\n### Features (123 total)\n\n| Group | Features | Description |\n|-------|----------|-------------|\n| Hardware | GPU family, driver versions, APU/iGPU flags | From report system info |\n| Temporal | Report age | Days since report submission |\n| Game metadata | Engine, genre, DRM, anticheat, Deck status | From Steam, PCGamingWiki, AWACY |\n| SVD embeddings | GPU (16-20d), Game (16-20d) | From co-occurrence matrices |\n| Text | Keywords, sentiment, note lengths | From user notes |\n| Text embeddings | Sentence-transformer SVD (32d) | From verdict notes |\n| Game aggregates | Customization rates, fault rates | Per-game community signals |\n| **IRT features** | Game difficulty, contributor strictness | From Item Response Theory |\n| **Error features** | Contributor consistency, game agreement | Per-annotator and per-game stats |\n| Proton variant | official/ge/experimental/native | Runner type (top Stage 2 feature) |\n\n### Label noise and IRT\n\nThe main challenge is subjective labeling on the tinkering/works_oob boundary (~15-20% noise). Different users have different standards — choosing GE-Proton is \"tinkering\" for strict users but \"works_oob\" for lenient ones.\n\n**IRT solution:** Fit a 1PL Item Response Theory model on contributor×game interactions:\n- **θ (theta):** per-contributor strictness — how likely they are to say \"tinkering\"\n- **d (difficulty):** per-(game, GPU family) objective difficulty\n- **P(tinkering) = σ(θ - d):** separates annotator bias from game reality\n\nIRT features are the #2 and #4 most important features in Stage 2. Combined with contributor-aware relabeling, IRT provides +0.047 F1 improvement.\n\n### Experiment history\n\n50+ experiments across Phases 1-22. See [docs/](docs/) for detailed plans and results.\n\n#### Foundation (Phases 1-8)\n\n| Phase | Approach | Result |\n|---|---|---|\n| 1-3 | Python foundation, worker, preprocessing pipeline | Infrastructure |\n| 4 | Single LightGBM model → cascade (borked/works → tinkering/oob) | F1=0.593 → cascade +0.009 |\n| 5 | SVD embeddings (GPU×Game, CPU×Game co-occurrence) | +0.010 F1 |\n| 6 | Game metadata enrichment (Steam, PCGamingWiki, AWACY, GitHub) | +0.024 F1 |\n| 7 | Text features (keyword regex, sentiment, note lengths) | +0.008 F1 |\n| 8 | Rule-based relabeling (tinkering→oob if no effort markers) | +0.010 F1 |\n\n#### Noise reduction (Phases 9-10)\n\n| Phase | Approach | ΔF1 | Outcome |\n|---|---|---|---|\n| 9.1 | Label smoothing α=0.15 (cross_entropy objective) | +0.008 | Noise-robust training |\n| 9.2 | Per-game aggregate features (26 features) | +0.024 | Community signals |\n| 9.3 | Cleanlab noise removal (3% of train) | +0.021 | Confident learning |\n| 9.4 | Ordinal classification, distillation, focal loss | 0.000 | All negative (6 experiments) |\n| 9.5 | Feature combinations, target encoding | −0.002 | All negative (8 experiments) |\n| 10 | Text embeddings (sentence-transformers SVD 32d) | +0.005 | Partial text coverage (32%) |\n\n#### IRT breakthrough (Phases 11-13) — main contribution\n\n| Phase | Approach | ΔF1 | Outcome |\n|---|---|---|---|\n| 11.2 | Alternative models (CatBoost, XGBoost, HistGBM) | −0.002 | LightGBM optimal |\n| **12.8** | **IRT features (game difficulty + annotator strictness)** | **+0.030** | **Key innovation** |\n| 12.1-12.3 | Contributor features, sample weighting | −0.002 | IRT dominates |\n| 13.1 | IRT-only relabeling (replaces Phase 8 + Cleanlab) | +0.013 | Simpler, better |\n| **13.2** | **Contributor-aware relabeling by annotator θ** | **+0.017** | **Graduated relabeling** |\n| 13.3 | Hybrid pipeline, confidence weighting | −0.001 | Weighting hurts |\n| 13.4-13.5 | Iterative IRT, annotator SVD embeddings | +0.001 | Marginal |\n\n#### Optimization attempts (Phases 14-19) — mostly negative\n\n| Phase | Approach | ΔF1 | Outcome |\n|---|---|---|---|\n| 14 | Steam PICS features (runtime, deck tests, review) | 0.000 | Redundant with existing |\n| 15 | Temporal features, Proton×Game SVD, Factorization Machines | −0.003..−0.017 | report_age_days sufficient |\n| **16** | **Class weight 1.8x oob + error features** | **+0.006** | **Compensate temporal shift** |\n| **17** | **HP tuning (reg=1.0) + ensemble** | **+0.004** | **Stronger regularization** |\n| 18 | Threshold optimization, focal loss, adaptive soft labels | 0.000 | Already calibrated |\n| 19 | LLM verdict inference (OpenRouter), data filtering | −0.001 | IRT already optimal |\n\n#### Task reformulation (Phases 20-22) — production framing\n\n| Phase | Approach | ΔF1 | Outcome |\n|---|---|---|---|\n| 20 | Per-(game, gpu) aggregated evaluation | — | F1 0.780→0.829 per-pair |\n| **21** | **Per-game majority vote aggregation** | **+0.091** | **Production metric: F1=0.871** |\n| 21.5 | Aggregated model (trained on pairs) | — | Leakage confirmed |\n| 21.7 | Cold-start model (metadata only) | — | Binary F1=0.601 |\n| 22 | Text embeddings upgrade, Stage 1 tuning | 0.000 | Pipeline saturated |\n\n## Project structure\n\n```\nprotondb_settings/\n  api/          FastAPI server and routes\n  db/           SQLite connection, migrations\n  ml/\n    irt.py            IRT fitting, features, contributor-aware relabeling\n    aggregate.py      Per-game aggregated prediction\n    models/           Cascade classifier (Stage 1 + Stage 2)\n    features/         Feature engineering (embeddings, encoding, game aggregates)\n    train.py          Training pipeline\n    predict.py        Single-sample prediction\n    noise.py          Cleanlab noise detection (legacy, replaced by IRT)\n    relabeling.py     Rule-based relabeling (legacy, replaced by IRT)\n  preprocessing/\n    cleaning.py         Data cleaning\n    normalize/          GPU/CPU/driver normalization (heuristic + LLM)\n    enrichment/         External API data (Steam, Steam PICS, PCGamingWiki,\n                        ProtonDB, ProtonDB Reports, AWACY, GitHub)\n    llm/                LLM client (OpenAI, OpenRouter, Claude CLI backends)\n    extract/            Structured text extraction + verdict inference\n  worker/       ProtonDB dump fetcher and importer\n  cli.py        Click CLI entry point\n  config.py     Environment-based configuration\nscripts/        Experiment scripts (50+ experiments, reproducibility)\ndocs/           Architecture plans, ML experiment logs, research notes\n```\n\n## License\n\n[MIT](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgetjump%2Fprotondb-game-compatibility-prediction","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgetjump%2Fprotondb-game-compatibility-prediction","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgetjump%2Fprotondb-game-compatibility-prediction/lists"}