{"id":16449266,"url":"https://github.com/LPengYang/MotionClone","last_synced_at":"2025-10-27T06:31:06.067Z","repository":{"id":243794755,"uuid":"805367406","full_name":"LPengYang/MotionClone","owner":"LPengYang","description":"[ICLR 2025] Official implementation of MotionClone: Training-Free Motion Cloning for Controllable Video Generation ","archived":false,"fork":false,"pushed_at":"2025-01-03T11:57:21.000Z","size":60672,"stargazers_count":431,"open_issues_count":1,"forks_count":31,"subscribers_count":19,"default_branch":"main","last_synced_at":"2025-02-01T19:03:07.791Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LPengYang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-24T12:28:25.000Z","updated_at":"2025-02-01T18:10:13.000Z","dependencies_parsed_at":"2024-08-07T11:12:12.267Z","dependency_job_id":null,"html_url":"https://github.com/LPengYang/MotionClone","commit_stats":null,"previous_names":["bujiazi/motionclone","lpengyang/motionclone"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LPengYang%2FMotionClone","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LPengYang%2FMotionClone/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LPengYang%2FMotionClone/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LPengYang%2FMotionClone/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LPengYang","download_url":"https://codeload.github.com/LPengYang/MotionClone/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238445953,"owners_count":19473846,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T10:01:32.928Z","updated_at":"2025-10-27T06:30:59.057Z","avatar_url":"https://github.com/LPengYang.png","language":"Python","funding_links":[],"categories":["Python","Projects"],"sub_categories":["🎥 Video"],"readme":"# MotionClone\nThis repository is the official implementation of [MotionClone](https://arxiv.org/abs/2406.05338). It is a **training-free framework** that enables motion cloning from a reference video for controllable video generation, **without cumbersome video inversion processes**.\n\u003cdetails\u003e\u003csummary\u003eClick for the full abstract of MotionClone\u003c/summary\u003e\n\n\u003e Motion-based controllable video generation offers the potential for creating captivating visual content. Existing methods typically necessitate model training to encode particular motion cues or incorporate fine-tuning to inject certain motion patterns, resulting in limited flexibility and generalization.\nIn this work, we propose **MotionClone** a training-free framework that enables motion cloning from reference videos to versatile motion-controlled video generation, including text-to-video and image-to-video. Based on the observation that the dominant components in temporal-attention maps drive motion synthesis, while the rest mainly capture noisy or very subtle motions, MotionClone utilizes sparse temporal attention weights as motion representations for motion guidance, facilitating diverse motion transfer across varying scenarios. Meanwhile, MotionClone allows for the direct extraction of motion representation through a single denoising step, bypassing the cumbersome inversion processes and thus promoting both efficiency and flexibility. \nExtensive experiments demonstrate that MotionClone exhibits proficiency in both global camera motion and local object motion, with notable superiority in terms of motion fidelity, textual alignment, and temporal consistency.\n\u003c/details\u003e\n\n**[MotionClone: Training-Free Motion Cloning for Controllable Video Generation](https://arxiv.org/abs/2406.05338)** \n\u003c/br\u003e\n[Pengyang Ling*](https://github.com/LPengYang/),\n[Jiazi Bu*](https://github.com/Bujiazi/),\n[Pan Zhang\u003csup\u003e†\u003c/sup\u003e](https://panzhang0212.github.io/),\n[Xiaoyi Dong](https://scholar.google.com/citations?user=FscToE0AAAAJ\u0026hl=en/),\n[Yuhang Zang](https://yuhangzang.github.io/),\n[Tong Wu](https://wutong16.github.io/),\n[Huaian Chen](https://scholar.google.com.hk/citations?hl=zh-CN\u0026user=D6ol9XkAAAAJ),\n[Jiaqi Wang](https://myownskyw7.github.io/),\n[Yi Jin\u003csup\u003e†\u003c/sup\u003e](https://scholar.google.ca/citations?hl=en\u0026user=mAJ1dCYAAAAJ)  \n(*Equal Contribution)(\u003csup\u003e†\u003c/sup\u003eCorresponding Author)\n\n\u003c!-- [Arxiv Report](https://arxiv.org/abs/2307.04725) | [Project Page](https://animatediff.github.io/) --\u003e\n[![arXiv](https://img.shields.io/badge/arXiv-2406.05338-b31b1b.svg)](https://arxiv.org/abs/2406.05338)\n[![Project Page](https://img.shields.io/badge/Project-Website-green)](https://bujiazi.github.io/motionclone.github.io/)\n![](https://img.shields.io/github/stars/LPengYang/MotionClone?style=social)\n\u003c!-- [![Open in OpenXLab](https://cdn-static.openxlab.org.cn/app-center/openxlab_app.svg)](https://bujiazi.github.io/motionclone.github.io/) --\u003e\n\u003c!-- [![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-yellow)](https://bujiazi.github.io/motionclone.github.io/) --\u003e\n\n## Demo\n[![]](https://github.com/user-attachments/assets/d1f1c753-f192-455b-9779-94c925e51aaa)\n\n\n## 🖋 News\n- The latest version of our paper (**v4**) is available on arXiv! (10.08)\n- The latest version of our paper (**v3**) is available on arXiv! (7.2)\n- Code released! (6.29)\n\n## 🏗️ Todo\n- [x] We have updated the latest version of MotionCloning, which performs motion transfer **without video inversion** and supports **image-to-video and sketch-to-video**.\n- [x] Release the MotionClone code (We have released **the first version** of our code and will continue to optimize it. We welcome any questions or issues you may have and will address them promptly.)\n- [x] Release paper\n\n## 📚 Gallery\nWe show more results in the [Project Page](https://bujiazi.github.io/motionclone.github.io/).\n\n## 🚀 Method Overview\n### Feature visualization\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src='__assets__/feature_visualization.png'/\u003e\n\u003c/div\u003e\n\n### Pipeline\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src='__assets__/pipeline.png'/\u003e\n\u003c/div\u003e\n\nMotionClone utilizes sparse temporal attention weights as motion representations for motion guidance, facilitating diverse motion transfer across varying scenarios. Meanwhile, MotionClone allows for the direct extraction of motion representation through a single denoising step, bypassing the cumbersome inversion processes and thus promoting both efficiency and flexibility.\n\n## 🔧 Installations (python==3.11.3 recommended)\n\n### Setup repository and conda environment\n\n```\ngit clone https://github.com/Bujiazi/MotionClone.git\ncd MotionClone\n\nconda env create -f environment.yaml\nconda activate motionclone\n```\n\n## 🔑 Pretrained Model Preparations\n\n### Download Stable Diffusion V1.5\n\n```\ngit lfs install\ngit clone https://huggingface.co/runwayml/stable-diffusion-v1-5 models/StableDiffusion/\n```\n\nAfter downloading Stable Diffusion, save them to `models/StableDiffusion`. \n\n### Prepare Community Models\n\nManually download the community `.safetensors` models from [RealisticVision V5.1](https://civitai.com/models/4201?modelVersionId=130072) and save them to `models/DreamBooth_LoRA`. \n\n### Prepare AnimateDiff Motion Modules\n\nManually download the AnimateDiff modules from [AnimateDiff](https://github.com/guoyww/AnimateDiff), we recommend [`v3_adapter_sd_v15.ckpt`](https://huggingface.co/guoyww/animatediff/blob/main/v3_sd15_adapter.ckpt) and [`v3_sd15_mm.ckpt.ckpt`](https://huggingface.co/guoyww/animatediff/blob/main/v3_sd15_mm.ckpt). Save the modules to `models/Motion_Module`.\n\n### Prepare SparseCtrl for image-to-video and sketch-to-video\nManually download \"v3_sd15_sparsectrl_rgb.ckpt\" and \"v3_sd15_sparsectrl_scribble.ckpt\" from [AnimateDiff](https://huggingface.co/guoyww/animatediff/tree/main). Save the modules to `models/SparseCtrl`.\n\n## 🎈 Quick Start\n\n### Perform Text-to-video generation with customized camera motion\n```\npython t2v_video_sample.py --inference_config \"configs/t2v_camera.yaml\" --examples \"configs/t2v_camera.jsonl\"\n```\n### Perform Text-to-video generation with customized object motion\n```\npython t2v_video_sample.py --inference_config \"configs/t2v_object.yaml\" --examples \"configs/t2v_object.jsonl\"\n```\n### Combine motion cloning with sketch-to-video\n```\npython i2v_video_sample.py --inference_config \"configs/i2v_sketch.yaml\" --examples \"configs/i2v_sketch.jsonl\"\n```\n### Combine motion cloning with image-to-video\n```\npython i2v_video_sample.py --inference_config \"configs/i2v_rgb.yaml\" --examples \"configs/i2v_rgb.jsonl\"\n```\n\n\n## 📎 Citation \n\nIf you find this work helpful, please cite the following paper:\n\n```\n@article{ling2024motionclone,\n  title={MotionClone: Training-Free Motion Cloning for Controllable Video Generation},\n  author={Ling, Pengyang and Bu, Jiazi and Zhang, Pan and Dong, Xiaoyi and Zang, Yuhang and Wu, Tong and Chen, Huaian and Wang, Jiaqi and Jin, Yi},\n  journal={arXiv preprint arXiv:2406.05338},\n  year={2024}\n}\n```\n\n## 📣 Disclaimer\n\nThis is official code of MotionClone.\nAll the copyrights of the demo images and audio are from community users. \nFeel free to contact us if you would like remove them.\n\n## 💞 Acknowledgements\nThe code is built upon the below repositories, we thank all the contributors for open-sourcing.\n* [AnimateDiff](https://github.com/guoyww/AnimateDiff)\n* [FreeControl](https://github.com/genforce/freecontrol)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FLPengYang%2FMotionClone","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FLPengYang%2FMotionClone","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FLPengYang%2FMotionClone/lists"}