{"id":15167801,"url":"https://github.com/menyifang/MIMO","last_synced_at":"2025-10-01T00:31:58.998Z","repository":{"id":257802182,"uuid":"862421352","full_name":"menyifang/MIMO","owner":"menyifang","description":"Official implementation of \"MIMO: Controllable Character Video Synthesis with Spatial Decomposed Modeling\"","archived":false,"fork":false,"pushed_at":"2025-06-19T01:52:26.000Z","size":26785,"stargazers_count":1510,"open_issues_count":29,"forks_count":59,"subscribers_count":125,"default_branch":"main","last_synced_at":"2025-06-19T02:35:54.257Z","etag":null,"topics":["character-animation","diffusion-models","video-synthesis"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/menyifang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-09-24T15:12:47.000Z","updated_at":"2025-06-19T01:52:30.000Z","dependencies_parsed_at":null,"dependency_job_id":"e3ccfc13-f2b8-4ab7-8ea4-dd1e51a2cdf2","html_url":"https://github.com/menyifang/MIMO","commit_stats":null,"previous_names":["menyifang/mimo"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/menyifang/MIMO","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/menyifang%2FMIMO","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/menyifang%2FMIMO/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/menyifang%2FMIMO/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/menyifang%2FMIMO/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/menyifang","download_url":"https://codeload.github.com/menyifang/MIMO/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/menyifang%2FMIMO/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":277777968,"owners_count":25875397,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-30T02:00:09.208Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["character-animation","diffusion-models","video-synthesis"],"created_at":"2024-09-27T06:00:56.507Z","updated_at":"2025-10-01T00:31:58.992Z","avatar_url":"https://github.com/menyifang.png","language":"Python","funding_links":[],"categories":["Video Generation","\u003cspan id=\"video\"\u003eVideo\u003c/span\u003e"],"sub_categories":["\u003cspan id=\"tool\"\u003eLLM (LLM \u0026 Tool)\u003c/span\u003e"],"readme":"# MIMO - Official PyTorch Implementation\n\n### [Project page](https://menyifang.github.io/projects/MIMO/index.html) | [Paper](https://arxiv.org/abs/2409.16160) | [Video](https://www.youtube.com/watch?v=skw9lPKFfcE) | [Online Demo](https://modelscope.cn/studios/iic/MIMO)\n\n\u003e **MIMO: Controllable Character Video Synthesis with Spatial Decomposed Modeling**\u003cbr\u003e\n\u003e [Yifang Men](https://menyifang.github.io/), [Yuan Yao](mailto:yaoy92@gmail.com), [Miaomiao Cui](mailto:miaomiao.cmm@alibaba-inc.com), [Liefeng Bo](https://scholar.google.com/citations?user=FJwtMf0AAAAJ\u0026hl=en)\u003cbr\u003e\n\u003e Institute for Intelligent Computing (Tongyi Lab), Alibaba Group\n\u003e In: CVPR 2025 \n\nMIMO is a generalizable model for controllable video synthesis, which can not only synthesize realistic character videos with controllable attributes (i.e., character, motion and scene) provided by very simple user inputs, but also simultaneously achieve advanced scalability to arbitrary characters, generality to novel 3D motions, and applicability to interactive real-world scenes in a unified framework. \n\n## Demo\n\nAnimating character image with driving 3D pose from motion dataset\n\nhttps://github.com/user-attachments/assets/3a13456f-9ee5-437c-aba4-30d8c3b6e251\n\nDriven by in-the-wild video with spatial 3D motion and interactive scene\n\nhttps://github.com/user-attachments/assets/4d989e7f-a623-4339-b3d1-1d1a33ad25f2\n\n\nMore results can be found in [project page](https://menyifang.github.io/projects/MIMO/index.html).\n\n\n## 📢 News\n(2025-06-11) The code is released! We released a simplified version of full implementation, but it could achieve comparable performance.\n\n(2025-02-27) The paper is accepted by CVPR 2025! The full version of the paper is available on [arXiv](https://arxiv.org/abs/2409.16160).\n\n(2024-01-07) The online demo (v1.5) supporting custom driving videos is available now! Try out [![ModelScope Spaces](\nhttps://img.shields.io/badge/ModelScope-Spaces-blue)](https://modelscope.cn/studios/iic/MIMO).\n\n(2024-11-26) The online demo (v1.0) is available on ModelScope now! Try out [![ModelScope Spaces](\nhttps://img.shields.io/badge/ModelScope-Spaces-blue)](https://modelscope.cn/studios/iic/MIMO). The 1.5 version to support custom driving videos will be coming soon.\n\n(2024-09-25) The project page, demo video and technical report are released. The full paper version with more details is in process.\n\n\n\n## Requirements\n* python (\u003e=3.10)\n* pyTorch\n* tensorflow\n* cuda 12.1\n* GPU (tested on A100, L20)\n\n\n## 🚀 Getting Started\n\n```bash\ngit clone https://github.com/menyifang/MIMO.git\ncd MIMO\n```\n\n### Installation\n```bash\nconda create -n mimo python=3.10\nconda activate mimo\nbash install.sh\n```\n\n### Downloads\n\n#### Model Weights \n\nYou can manually download model weights from [ModelScope](https://modelscope.cn/models/iic/MIMO/files) or [Huggingface](https://huggingface.co/menyifang/MIMO/tree/main), or automatically using follow commands.\n\nDownload from HuggingFace\n```python\nfrom huggingface_hub import snapshot_download \nmodel_dir = snapshot_download(repo_id='menyifang/MIMO', cache_dir='./pretrained_weights')\n```\n\nDownload from ModelScope \n```python\nfrom modelscope import snapshot_download\nmodel_dir = snapshot_download(model_id='iic/MIMO', cache_dir='./pretrained_weights')\n```\n\n\n#### Prior Model Weights \n\nDownload pretrained weights of based model and other components: \n- [StableDiffusion V1.5](https://huggingface.co/runwayml/stable-diffusion-v1-5)\n- [sd-vae-ft-mse](https://huggingface.co/stabilityai/sd-vae-ft-mse)\n- [image_encoder](https://huggingface.co/lambdalabs/sd-image-variations-diffusers/tree/main/image_encoder)\n\n\n#### Data Preparation\n\nDownload examples and resources (`assets.zip`) from [google drive](https://drive.google.com/file/d/1dg0SDAxEARClYq_6L1T1XIfWvC5iA8WD/view?usp=drive_link) and unzip it under `${PROJECT_ROOT}/`.\nYou can also process custom videos following [Process driving templates](#process-driving-templates).\n\nAfter downloading weights and data, the folder of the project structure seems like:\n\n```text\n./pretrained_weights/\n|-- image_encoder\n|   |-- config.json\n|   `-- pytorch_model.bin\n|-- denoising_unet.pth\n|-- motion_module.pth\n|-- pose_guider.pth\n|-- reference_unet.pth\n|-- sd-vae-ft-mse\n|   |-- config.json\n|   |-- diffusion_pytorch_model.bin\n|   `-- diffusion_pytorch_model.safetensors\n`-- stable-diffusion-v1-5\n    |-- feature_extractor\n    |   `-- preprocessor_config.json\n    |-- model_index.json\n    |-- unet\n    |   |-- config.json\n    |   `-- diffusion_pytorch_model.bin\n    `-- v1-inference.yaml\n./assets/\n|-- video_template\n|   |-- template1\n\n```\n\nNote: If you have installed some of the pretrained models, such as `StableDiffusion V1.5`, you can specify their paths in the config file (e.g. `./config/prompts/animation_edit.yaml`).\n\n\n### Inference\n\n- video character editing\n```bash\npython run_edit.py\n```\n\n- character image animation\n```bash\npython run_animate.py\n```\n\n\n### Process driving templates\n\n- install external dependencies by\n```bash\nbash setup.sh\n```\nyou can also use dockerfile(`video_decomp/docker/decomp.dockerfile`) to build a docker image with all dependencies installed.\n\n\n- download model weights and data from [Huggingface](https://huggingface.co/menyifang/MIMO_VidDecomp/tree/main) and put them under `${PROJECT_ROOT}/video_decomp/`.\n\n```python\nfrom huggingface_hub import snapshot_download \nmodel_dir = snapshot_download(repo_id='menyifang/MIMO_VidDecomp', cache_dir='./video_decomp/')\n```\n\n\n- process the driving video by\n```bash\ncd video_decomp\npython run.py\n```\n\nThe processed template can be putted under `${PROJECT_ROOT}/assets/video_template` for editing and animation tasks as follows:\n```\n./assets/video_template/\n|-- template1/\n|   |-- vid.mp4\n|   |-- mask.mp4\n|   |-- sdc.mp4\n|   |-- bk.mp4\n|   |-- occ.mp4 (if existing)\n|-- template2/\n|-- ...\n|-- templateN/\n```\n\n### Training\n\n\n\n## 🎨 Gradio Demo\n\n**Online Demo**: We launch an online demo of MIMO at [ModelScope Studio](https://modelscope.cn/studios/iic/MIMO).\n\nIf you have your own GPU resource (\u003e= 40GB vram), you can run a local gradio app via following commands:\n\n`python app.py`\n\n\n\n## Acknowledgments\n\nThanks for great work from [Moore-AnimateAnyone](https://github.com/MooreThreads/Moore-AnimateAnyone), [SAM](https://github.com/facebookresearch/segment-anything), [4D-Humans](https://github.com/shubham-goel/4D-Humans), [ProPainter](https://github.com/sczhou/ProPainter)\n\n\n## Citation\n\nIf you find this code useful for your research, please use the following BibTeX entry.\n\n```bibtex\n@inproceedings{men2025mimo,\n  title={MIMO: Controllable Character Video Synthesis with Spatial Decomposed Modeling},\n  author={Men, Yifang and Yao, Yuan and Cui, Miaomiao and Liefeng Bo},\n  booktitle={Computer Vision and Pattern Recognition (CVPR), 2025 IEEE Conference on},\n  year={2025}}\n}\n```\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmenyifang%2FMIMO","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmenyifang%2FMIMO","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmenyifang%2FMIMO/lists"}