{"id":29009895,"url":"https://github.com/tencentarc/animegamer","last_synced_at":"2025-10-10T23:46:35.469Z","repository":{"id":284898292,"uuid":"956418076","full_name":"TencentARC/AnimeGamer","owner":"TencentARC","description":"AnimeGamer: Infinite Anime Life Simulation with Next Game State Prediction","archived":false,"fork":false,"pushed_at":"2025-04-09T07:55:41.000Z","size":45684,"stargazers_count":318,"open_issues_count":6,"forks_count":27,"subscribers_count":15,"default_branch":"main","last_synced_at":"2025-06-25T15:51:54.533Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://howe125.github.io/AnimeGamer.github.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TencentARC.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"License.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-03-28T08:15:35.000Z","updated_at":"2025-06-23T07:59:46.000Z","dependencies_parsed_at":"2025-06-25T15:45:51.913Z","dependency_job_id":null,"html_url":"https://github.com/TencentARC/AnimeGamer","commit_stats":null,"previous_names":["tencentarc/animegamer"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/TencentARC/AnimeGamer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TencentARC%2FAnimeGamer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TencentARC%2FAnimeGamer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TencentARC%2FAnimeGamer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TencentARC%2FAnimeGamer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TencentARC","download_url":"https://codeload.github.com/TencentARC/AnimeGamer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TencentARC%2FAnimeGamer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279005583,"owners_count":26083921,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-10T02:00:06.843Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-25T15:33:35.593Z","updated_at":"2025-10-10T23:46:35.454Z","avatar_url":"https://github.com/TencentARC.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AnimeGamer: Infinite Anime Life Simulation with Next Game State Prediction\n\n**[Junhao Cheng\u003csup\u003e1,2\u003c/sup\u003e](https://donahowe.github.io/), \n[Yuying Ge\u003csup\u003e1,\u0026#9993;\u003c/sup\u003e](https://geyuying.github.io/), \n[Yixiao Ge\u003csup\u003e1\u003c/sup\u003e](https://geyixiao.com/), \n[Jing Liao\u003csup\u003e2\u003c/sup\u003e](https://scholar.google.com/citations?user=3s9f9VIAAAAJ\u0026hl=en), \n[Ying Shan\u003csup\u003e1\u003c/sup\u003e](https://scholar.google.com/citations?user=4oXBp9UAAAAJ\u0026hl=en)**\n\u003cbr\u003e\n\u003csup\u003e1\u003c/sup\u003eARC Lab, Tencent PCG, \n\u003csup\u003e2\u003c/sup\u003eCity University of Hong Kong\n\u003cbr\u003e\n\n[![Project Page](https://img.shields.io/badge/Project-blue)](https://howe125.github.io/AnimeGamer.github.io/)\n[![arXiv](https://img.shields.io/badge/arXiv-2404.14396-b31b1b.svg)](https://arxiv.org/abs/2504.01014)\n[![Static Badge](https://img.shields.io/badge/Model-Huggingface-yellow)](https://huggingface.co/TencentARC/AnimeGamer)\n\n\n## 🔎 Introduction\n**Experience the endless adventure of infinite anime life with AnimeGamer!** 🤩\n![teaser](assets/Intro.gif)\n\nYou can step into the shoes of Sosuke from \"Ponyo on the Cliff\" and interact with a dynamic game world through open-ended language instructions. AnimeGamer generates consistent multi-turn game states, consisting of dynamic animation shots (i.e., videos ) with contextual consistency (e.g., the purple car\nand the forest background), and updates to character states including stamina, social, and entertainment values.\n\n![teaser](assets/Intro2.gif)\nWith AnimeGamer, you can bring together beloved characters like Qiqi from \"Qiqi's Delivery Service\" and Pazu from \"Castle in the Sky\" to meet and interact in the anime world. Imagine Pazu mastering Qiqi's broom-flying skills, creating unique and magical experiences. AnimeGamer can generalize interactions between characters from different anime films and character actions, with the potential for endless possibilities.\n\n## :book: Method\n\n![teaser](assets/model.png)\n\nAnimeGamer is built upon Multimodal Large Language Models (MLLMs) to generate each game state, including dynamic animation shots that depict character movements and updates to character states. The overview of AnimeGamer is as follows. The training process consists of three phases:\n* (a) We model animation shots using action-aware multimodal representations through an encoder and train a diffusion-based decoder to reconstruct videos, with the additional input of motion scope that indicates action intensity. \n* (b) We train an MLLM to predict the next game state representations by taking the history instructions and game state representations as input.\n* (c) We further enhance the quality of decoded animation shots from the MLLM via an adaptation phase, where the decoder is fine-tuned by taking MLLM's predictions as input.\n\n\n\n## 📅 News\n\n* [2025-04-09] Release local gradio demo (interactive generation)🔥🔥🔥\n* [2025-04-02] Release wights of models separately trained on \"Qiqi's Delivery Service\" and \"Ponyo on the cliff\" 🔥\n* [2025-04-02] Release paper in [arXiv](https://arxiv.org/abs/2504.01014) 🔥🔥🔥\n* [2025-04-01] Release inference codes 🔥🔥🔥\n* [2025-03-28] Create the repository 🔥🔥🔥\n\n\n## 🔜 TODOs\n- [ ] Release data processing pipeline\n- [ ] Release training codes \n- [ ] Release wights of models trained on a mixture of anime films (the same setting as in our paper)\n\n\n\n\n\n## 🚀 Quick Start\n\n### 🔮 Environment Setup\n\nTo set up the environment for inference, you can run the following command:\n```shell\ngit clone https://github.com/TencentARC/AnimeGamer.git\ncd AnimeGamer\nconda create -n animegamer python==3.10 -y\nconda activate animegamer\npip install -r requirements.txt\n```\n\nPlease first download the checkpoints of [AnimeGamer](https://huggingface.co/TencentARC/AnimeGamer) and [Mistral-7B](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1), and save them under the folder `./checkpoints`.\nThen you should download the 3D-VAE of CogvideoX:\n```shell\ncd checkpoints\nwget https://cloud.tsinghua.edu.cn/f/fdba7608a49c463ba754/?dl=1\nmv 'index.html?dl=1' vae.zip\nunzip vae.zip\n```\n### 🧸 Gradio \nTo run local gradio demo:\n```shell\npython app.py \n```\nThis Gradio demo is designed for low VRAM, where the MLLM and the VDM Decoder are deployed on two GPUs (each with at least 24G VRAM). If you wish to deploy on a single GPU (at least 60G VRAM), please set `LOW_VRAM_VERSION = False`.\n\n### 🪄 Inference\nTo generate action-aware multimodal representations and update character states, you can run:\n```shell\npython inference_MLLM.py \n```\n\nTo decode the representations into animation shots, you can run:\n```shell\npython inference_Decoder.py \n```\n\nChange the instructions in `./game_demo` to customize your play.\n\n\n\n## 🤗 Acknowledgements\n\nWe refer to [CogvideoX](https://github.com/XLabs-AI/x-flux) and [SEED-X](https://github.com/AILab-CVC/SEED-X/tree/main) to build our codebase. Thanks for their wonderful project.\n\n\n\n## 📜 Citation\n\nIf you find this work helpful, please consider citing:\n\n```BibTeXw\n@article{cheng2025animegamer,\n  title={AnimeGamer: Infinite Anime Life Simulation with Next Game State Prediction},\n  author={Cheng, Junhao and Ge, Yuying and Ge, Yixiao and Liao, Jing and Shan, Ying},\n  journal={arXiv preprint arXiv:2504.01014},\n  year={2025}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftencentarc%2Fanimegamer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftencentarc%2Fanimegamer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftencentarc%2Fanimegamer/lists"}