{"id":51602704,"url":"https://github.com/freedomintelligence/gamecraft-bench","last_synced_at":"2026-07-11T23:01:23.996Z","repository":{"id":365262600,"uuid":"1271267166","full_name":"FreedomIntelligence/gamecraft-bench","owner":"FreedomIntelligence","description":"Code and Data for paper \"GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?\"","archived":false,"fork":false,"pushed_at":"2026-07-02T08:22:14.000Z","size":3451,"stargazers_count":169,"open_issues_count":1,"forks_count":10,"subscribers_count":6,"default_branch":"main","last_synced_at":"2026-07-02T10:16:28.083Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://tongxuluo.github.io/gamecraft-bench-website/","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FreedomIntelligence.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-16T13:49:46.000Z","updated_at":"2026-07-02T08:22:18.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/FreedomIntelligence/gamecraft-bench","commit_stats":null,"previous_names":["tongxuluo/gamecraft-bench","freedomintelligence/gamecraft-bench"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/FreedomIntelligence/gamecraft-bench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2Fgamecraft-bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2Fgamecraft-bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2Fgamecraft-bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2Fgamecraft-bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FreedomIntelligence","download_url":"https://codeload.github.com/FreedomIntelligence/gamecraft-bench/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2Fgamecraft-bench/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35377013,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-11T02:00:05.354Z","response_time":104,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-11T23:01:22.908Z","updated_at":"2026-07-11T23:01:23.977Z","avatar_url":"https://github.com/FreedomIntelligence.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"media/benchmark_title.png\" alt=\"GameCraft-Bench\" width=\"90%\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://tongxuluo.github.io/gamecraft-bench-website\"\u003eWebsite\u003c/a\u003e •\n  \u003ca href=\"https://arxiv.org/abs/2606.17861\"\u003ePaper\u003c/a\u003e •\n  \u003ca href=\"https://tongxuluo.github.io/gamecraft-bench-website/#demos\"\u003eDemos\u003c/a\u003e\n\u003c/p\u003e\n\n**GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?**\n\nGameCraft-Bench evaluates whether coding agents can transform natural-language game specifications into complete, playable Godot projects.\nUnlike traditional coding tasks, game generation depends on scripts, scenes, assets, rendering, runtime configuration, and player-game interaction working together as one executable system.\n\nThe benchmark contains **140 tasks** across **15 game families**.\nEach agent must submit a complete Godot project together with replayable demonstration traces.\nThe verifier launches the project, replays the traces, records gameplay evidence, and scores observed play with a hidden rubric and multimodal judge.\n\nThe benchmark runs on top of [Harbor](https://github.com/harbor-framework/harbor) and ships with a custom local-subprocess environment for Docker-less hosts.\n\n---\n\n## Benchmark overview\n\nGameCraft-Bench is organized around three desiderata for end-to-end game generation:\n\n- **Engine Grounding**: games are generated and evaluated inside a concrete game engine and runtime environment.\n- **Artifact Completeness**: agents must deliver complete launchable game projects rather than isolated scripts, scenes, or assets.\n- **Interactive Verification**: games are judged by observed behavior under player input, using replayed demonstrations as standardized gameplay evidence.\n\nThe task suite covers diverse 2D game-generation demands, including continuous control and collision, rule and state management, progression and economy, exploration, narrative interaction, and presentation-heavy gameplay.\n\n| Family | Tasks | Family | Tasks | Family | Tasks |\n|---|---:|---|---:|---|---:|\n| Platformer | 19 | Strategy | 17 | Tycoon | 16 |\n| Open-world | 15 | Roguelike | 14 | Visual novel | 11 |\n| Puzzle | 8 | Shooter | 7 | Simulation | 6 |\n| Card game | 5 | Horror | 5 | Rhythm | 5 |\n| Idle | 4 | Racing | 4 | Sports | 4 |\n\n## Main results\n\nFrontier coding agents remain far from reliable end-to-end game generation.\nThe strongest evaluated configuration reaches only **41.46%** overall, and most agents score below 40%.\n\n| Harness | Model | Overall | Mechanics | Depth | Visuals | Art |\n|---|---|---:|---:|---:|---:|---:|\n| Claude Code | Opus-4.7 high | **41.46** | **55.34** | **39.48** | **42.78** | **36.86** |\n| Codex | GPT-5.5 high | 39.49 | 54.36 | 38.61 | 41.84 | 32.94 |\n| Kimi Code | Kimi-K2.6 | 30.65 | 39.76 | 28.07 | 33.66 | 27.99 |\n| Claude Code | MiMo-V2.5-Pro | 24.10 | 32.33 | 22.59 | 27.45 | 20.65 |\n| Code Buddy | GLM-5.1 | 18.29 | 25.23 | 17.80 | 21.14 | 14.59 |\n| Code Buddy | MiniMax-M2.7 | 10.95 | 14.27 | 9.92 | 14.92 | 8.85 |\n| Codex | DeepSeek-V4-Pro | 2.15 | 2.25 | 1.69 | 1.97 | 2.63 |\n\nScores are percentages. Mechanics, Depth, Visuals, and Art correspond to the\nfour rubric categories: Core Mechanics, Content Depth, Functional Visuals, and\nArt and Presentation.\n\n## Token usage\n\nFor the model runs listed below, we report token usage rather than dollar cost because provider pricing, routing endpoints, cache policies, and enterprise agreements vary. Users can combine these counts with their own provider pricing to estimate monetary cost.\n\n| Model | Tasks | Input tokens total | Cache tokens total | Output tokens total | Avg input / task | Avg output / task |\n|---|---:|---:|---:|---:|---:|---:|\n| Kimi-K2.6 | 140 | 2.24B | 2.22B | 10.9M | 16.0M | 77.7K |\n| Claude Opus-4.7 | 140 | 1.58B | 1.50B | 9.4M | 11.3M | 67.0K |\n| DeepSeek-V4-Pro | 140 | 232.6M | 58.0M | 7.8M | 1.7M | 55.4K |\n| GPT-5.5 (Codex) | 140 | 148.5M | 130.3M | 2.8M | 1.1M | 20.0K |\n\n## Evaluation protocol\n\nEach task provides a natural-language game specification, a Godot-based workspace, shared resources, and a hidden rubric.\nA valid submission includes a Godot project under `/workspace/game` and replayable input traces under `/workspace/game/demo_outputs/`.\nThe verifier checks launchability, replays the submitted traces, records gameplay evidence, and applies the hidden rubric to the observed behavior.\n\n## Install\n\nTested on Ubuntu 22.04. Run as root (or with `sudo`).\n\n### 1. System dependencies\n\n```bash\napt update\nDEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \\\n    xvfb xdotool ffmpeg x11-utils x11-xserver-utils \\\n    libxcursor1 libxinerama1 libxrandr2 libxi6 libgl1 libegl1 \\\n    unzip ca-certificates curl \\\n    x11vnc novnc\n```\n\nThese cover everything the verifier and dashboard need:\n- `xvfb` — virtual X display for headless replay\n- `xdotool` — synthetic mouse / keyboard injection\n- `ffmpeg` — screen recording (`x11grab`)\n- `x11-utils` / `x11-xserver-utils` — display probes\n- `libxcursor1` … `libegl1` — runtime libraries Godot dlopens\n- `x11vnc` / `novnc` — browser-based dashboard play sessions\n\n### 2. Godot 4.6.2\n\nPinned for reproducibility. Direct GitHub download often fails behind the GFW; the `gh-proxy.com` mirror is the easy fallback:\n\n```bash\nmkdir -p /opt/godot \u0026\u0026 cd /opt/godot\ncurl -sSL -o godot.zip \\\n    \"https://gh-proxy.com/https://github.com/godotengine/godot/releases/download/4.6.2-stable/Godot_v4.6.2-stable_linux.x86_64.zip\"\nunzip -o godot.zip \u0026\u0026 rm godot.zip\nmv Godot_v4.6.2-stable_linux.x86_64 godot\nchmod +x godot\nln -sf /opt/godot/godot /usr/local/bin/godot\n\ngodot --version       # → 4.6.2.stable.official.71f334935\n```\n\nIf the direct link works for you, drop the `gh-proxy.com/` prefix.\n\n### 3. Python\n\n```bash\ngit clone \u003cthis-repo\u003e game-bench \u0026\u0026 cd game-bench\nuv venv --python 3.12 .venv\nsource .venv/bin/activate\nuv pip install -e .          # add --index-url https://pypi.tuna.tsinghua.edu.cn/simple if needed\n```\n\n### 4. Local config\n\n```bash\ncp .env.example .env         # fill in API keys / paths if defaults aren't right\n```\n\n`.env` holds judge API keys (`OPENAI_API_KEY`, `ANTHROPIC_AUTH_TOKEN`, …),\nthe path to the Godot binary, and any path overrides. All scripts under `scripts/` source it automatically.\n\n### 5. Asset libraries (optional but expected)\n\nTasks share two CC0 / permissive 2D asset pools, mounted read-only into each trial:\n\n- **Kenney** at `/workspace/assets/library/` — themed packs (sprites, tilesets, UI, audio). ~480 MB across ~157 packs.\n- **OpenGameArt** at `/workspace/assets/library-oga/` — narrative / RPG / pixel-art entries scraped from opengameart.org. Per-entry `LICENSE.txt` records the source URL and license string.\n\n```bash\npython scripts/fetch_assets.py                              # Kenney\npython scripts/fetch_oga_assets.py --license CC0            # OGA, CC0 only\n```\n\nBoth fetchers are idempotent; safe to re-run.\n\n## Run a task\n\n```bash\n./scripts/run.sh -p tasks/\u003ctask\u003e --agent \u003cagent\u003e\n```\n\n`run.sh` is a thin wrapper around `harbor run` that sources `.env`, activates the venv, sets `PYTHONPATH`, and points harbor at our custom\n`LocalSubprocessEnvironment`. Use `--agent oracle` to run the bundled reference solution; `--agent nop` to confirm a do-nothing agent fails (0.0).\nAll extra arguments are forwarded to Harbor, so `-n`, `-x`, `--job-name`, `--delete`, and `--ak ...` work the same way as in `harbor run`.\n\nFor local coding-agent runs, use the dedicated wrappers.\nThey source `.env`, pin the local agent implementation, forward the required API credentials into the agent subprocess, and default to `--no-delete` so generated projects remain under the job directory for inspection.\n\n```bash\n# Claude Code + Opus-4.7\n./scripts/run_claude_code_opus_4_7.sh --ak reasoning_effort=high -p tasks/\u003ctask\u003e\n\n# Codex CLI + GPT-5.5\n./scripts/run_codex_gpt_5_5.sh --ak reasoning_effort=high -p tasks/\u003ctask\u003e\n\n# Kimi Code + Kimi coding model\n./scripts/run_kimi_code.sh -p tasks/\u003ctask\u003e\n\n# Claude Code + MiMo-V2.5-Pro through an Anthropic-compatible endpoint\n./scripts/run_claude_code_mimo_2_5.sh --ak reasoning_effort=high -p tasks/\u003ctask\u003e\n```\n\nClaude Code / Codex-based wrappers require `--ak reasoning_effort=\u003clow|medium|high\u003e`.\nThe Kimi wrapper passes `--ak thinking=true` by default.\n\nJob artifacts land under `$GAMECRAFT_BENCH_JOBS_ROOT` (default `../gamecraft-bench-jobs/\u003ctimestamp\u003e/\u003ctask\u003e__\u003cid\u003e/`).\n\n## Dashboard\n\n`gamecraft_bench/dashboard/` is a browser-based dashboard for inspecting benchmark jobs, scores, artifacts, and the playable games agents produce.\nWhen launching a game for interactive inspection, it snapshots the project to a temporary play directory so the agent's retained sandbox is not disturbed.\n\n```bash\n./scripts/dashboard_service.sh             # default port 6090, default jobs root\n./scripts/dashboard_service.sh --port 7000 --jobs-root /custom/path\n```\n\nForward the port in VS Code (Ports panel), open `http://localhost:6090/`, then either:\n\n- Click **Play** on a single trial, or\n- Tick the checkboxes on multiple trials and click **Compare** to open a grid view with one live noVNC iframe per game (auto-laid-out 2/3/4 columns). Each cell has its own Refresh / Stop button and is fully interactive.\n\nArchitecture: a session pool of up to 8 X displays (`:300`–`:307`, disjoint from the verifier's `:99`–`:199` range), each backing a dedicated Xvfb + Godot + x11vnc trio. The FastAPI app serves noVNC's static files at `/novnc/` and bridges browser WebSocket frames to x11vnc TCP at `/ws/{sid}`. Closing the browser tab fires `navigator.sendBeacon` to free the slot.\n\n## Adding a task\n\n1. `harbor task init \u003corg\u003e/\u003ctask-name\u003e` from the repo root.\n2. Move the generated dir under `tasks/`.\n3. Fill in `instruction.md` and `tests/rubric.json` (build_check, requirements, score_formula).\n4. Optional: write `solution/solve.sh` for the oracle agent.\n5. End-to-end check:\n   ```bash\n   ./scripts/run.sh -p tasks/\u003ctask\u003e --agent oracle\n   ```\n\nTask assets / starter scaffold go under `tasks/\u003ctask\u003e/workspace/`; contents are copied into the agent's `/workspace/` at env start.\n\n## Scope\n\nGameCraft-Bench focuses on 2D game generation in Godot.\nThis makes the benchmark lightweight and reproducible for headless evaluation, but it does not cover Unity, Unreal, 3D games, multiplayer systems, large-scale physics, or long-form production workflows.\nThe verifier scores visual gameplay evidence; audio-dependent aspects are represented through visible game behavior rather than direct audio evaluation.\n\nThe benchmark measures whether an agent follows the game specification and realizes the requested mechanics, content, visual state, and presentation in an executable artifact.\nIt does not attempt to measure whether a generated game is subjectively fun.\n\n## Acknowledgment\n\nGameCraft-Bench builds on [Godot](https://github.com/godotengine/godot) as the game engine runtime and [Harbor](https://github.com/harbor-framework/harbor) as the benchmark and agent-execution harness.\nWe thank the open-source communities behind these projects for making reproducible, end-to-end game-generation evaluation possible.\n\n## Citation\n\nIf you find GameCraft-Bench useful, please cite us:\n\n```bibtex\n@article{luo2026gamecraft,\n  title={GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?},\n  author={Luo, Tongxu and Wang, Rongsheng and Bi, Jiaxi and Xu, Chenming and Tang, Zhengyang and Chen, Jianlong and Liang, Juhao and Ji, Ke and Guo, Shuqi and Du, Yuhao and Bu, Fan and Du, Wenyu and Zhang, Xiaotong and Li, Kyle and Wang, Shaobo and Zhang, Linfeng and Liu, Yuxuan and Lai, Xin and Li, Chenxin and Guo, Yiduo and Zhang, Zhexin and Wang, Xinyuan and Bai, Tianyi and Li, Ziniu and Wang, Benyou},\n  journal={arXiv preprint arXiv:2606.17861},\n  year={2026}\n}\n```\n\n## License\n\nApache-2.0.\n\n## Star History\n\n\u003ca href=\"https://www.star-history.com/?type=date\u0026repos=tongxuluo%2Fgamecraft-bench\"\u003e\n \u003cpicture\u003e\n   \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"https://api.star-history.com/chart?repos=tongxuluo/gamecraft-bench\u0026type=date\u0026theme=dark\u0026legend=top-left\" /\u003e\n   \u003csource media=\"(prefers-color-scheme: light)\" srcset=\"https://api.star-history.com/chart?repos=tongxuluo/gamecraft-bench\u0026type=date\u0026legend=top-left\" /\u003e\n   \u003cimg alt=\"Star History Chart\" src=\"https://api.star-history.com/chart?repos=tongxuluo/gamecraft-bench\u0026type=date\u0026legend=top-left\" /\u003e\n \u003c/picture\u003e\n\u003c/a\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Fgamecraft-bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffreedomintelligence%2Fgamecraft-bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Fgamecraft-bench/lists"}