{"id":13563077,"url":"https://github.com/diffusion-motion-transfer/diffusion-motion-transfer","last_synced_at":"2025-04-03T19:32:20.006Z","repository":{"id":212300315,"uuid":"728667944","full_name":"diffusion-motion-transfer/diffusion-motion-transfer","owner":"diffusion-motion-transfer","description":"Official Pytorch Implementation for \"Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer\"\"","archived":false,"fork":false,"pushed_at":"2023-12-13T12:58:35.000Z","size":9550,"stargazers_count":149,"open_issues_count":3,"forks_count":16,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-11-04T15:52:20.418Z","etag":null,"topics":["cvpr2024","diffusion-models","motion-transfer","space-time","spatio-temporal","text2video","video-editing"],"latest_commit_sha":null,"homepage":"https://diffusion-motion-transfer.github.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/diffusion-motion-transfer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-12-07T12:41:42.000Z","updated_at":"2024-10-30T05:58:55.000Z","dependencies_parsed_at":"2023-12-13T15:34:33.391Z","dependency_job_id":null,"html_url":"https://github.com/diffusion-motion-transfer/diffusion-motion-transfer","commit_stats":null,"previous_names":["diffusion-motion-transfer/diffusion-motion-transfer"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/diffusion-motion-transfer%2Fdiffusion-motion-transfer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/diffusion-motion-transfer%2Fdiffusion-motion-transfer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/diffusion-motion-transfer%2Fdiffusion-motion-transfer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/diffusion-motion-transfer%2Fdiffusion-motion-transfer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/diffusion-motion-transfer","download_url":"https://codeload.github.com/diffusion-motion-transfer/diffusion-motion-transfer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247065406,"owners_count":20877768,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cvpr2024","diffusion-models","motion-transfer","space-time","spatio-temporal","text2video","video-editing"],"created_at":"2024-08-01T13:01:14.908Z","updated_at":"2025-04-03T19:32:19.285Z","avatar_url":"https://github.com/diffusion-motion-transfer.png","language":"Python","funding_links":[],"categories":["Python","Poster"],"sub_categories":[],"readme":"# Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer\n\u003ca href=\"https://diffusion-motion-transfer.github.io/\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=Project\u0026message=Website\u0026color=blue\"\u003e\u003c/a\u003e\n\u003ca href=\"https://arxiv.org/abs/\"\u003e\u003cimg src=\"https://img.shields.io/badge/arXiv-2311.17009-b31b1b.svg\"\u003e\u003c/a\u003e\n\nThis is the official implementation of the paper:\n\n[**Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer**](https://diffusion-motion-transfer.github.io/)\n\u003cbr/\u003e\n\n[Danah Yatim*](https://www.linkedin.com/in/danah-yatim-4b15231b5/),\n[Rafail Fridman*](https://www.linkedin.com/in/rafail-fridman/),\n[Omer Bar-Tal](https://omerbt.github.io/),\n[Yoni Kasten](https://ykasten.github.io/),\n[Tali Dekel](https://www.weizmann.ac.il/math/dekel/)\n\u003cbr/\u003e\n(*equal contribution)\n\nhttps://github.com/diffusion-motion-transfer/diffusion-motion-transfer/assets/22198039/4fe912d4-0975-4580-af7f-19fd73b0cbfe\n\n\n\nIntroducing a zero-shot method for transferring motion across objects and scenes. without any training or finetuning.\n\n\u003eWe present a new method for text-driven motion transfer -- synthesizing a video that complies with an input text prompt describing the target objects and scene while maintaining an input video's motion and scene layout. Prior methods are confined to transferring motion across two subjects within the same or closely related object categories and are applicable for limited domains (e.g., humans). \nIn this work, we consider a significantly more challenging setting in which the target and source objects differ drastically in shape and fine-grained motion characteristics (e.g., translating a jumping dog into a dolphin).  To this end, we leverage a pre-trained and fixed text-to-video diffusion model, which provides us with generative and motion priors. The pillar of our method is a new space-time feature loss derived directly from the model. This loss guides the generation process to preserve the overall motion of the input video while complying with the target object in terms of shape and fine-grained motion traits. \n\nFor more, visit the [project webpage](https://diffusion-motion-transfer.github.io/).\n\n# Installation\nClone the repo and create a new environment:\n```\ngit clone https://github.com/diffusion-motion-transfer/diffusion-motion-transfer.git\ncd diffusion-motion-transfer\nconda create --name dmt python=3.9\nconda activate dmt\n```\nInstall our environment requirements:\n```\nconda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia\npip install -r requirements.txt\n```\n\n# Motion Transfer\n* Our method is designed for transferring motion across objects and scenes\n* Our method is based on [ZeroScope](https://huggingface.co/cerspense/zeroscope_v2_576w) text-to-video model. Therefore, we can edit videos of **24 frames**.\n* in some cases the combination of target object and input video motion is out of distribution for the T2V model, which can lead to visual artifacts in the generated video. It may be necessary to sample several seeds.\n* Method was tested to run on a single NVIDIA A40 48GB, and takes ~32GB of video memory. It takes approximately 7 minutes on a single NVIDIA A40 48GB.\n\n# Preprocess\nTo preprocess a video, update configuration file `configs/preprocess_config.yaml':\n\nArguments to update:\n* ```video_path``` - the input video frames should be located in this path\n* ```save_dir``` - the latents will be saved in this path\n* ```prompt``` - empty string or a string describing the video content\n\nOptional arguments to update:\n* ```--save_ddim_reconstruction``` if True, the reconstructed video will be saved in ```--save_dir```\n\nAfter updating config file, run the following command:\n```\npython preprocess_video_ddim.py --config_path configs/preprocess_config.yaml\n```\nOnce the preprocessing is done, the latents will be saved in the ```save_dir``` path. \n\n# Editing\nTo edit the video, update configuration file `configs/guidance_config.yaml`\nArguments to update:\n* ```data_path``` - the input video frames should be located in this path\n* ```output_path``` - the edited video will be saved in this path\n* ```latents_path``` - the latents of the input video should be located in this path\n* ```source_prompt``` - prompt used for inversion\n* ```target_prompt``` - prompt used for editing\n    \nOptional arguments to update:\n* ```negative_prompt``` - prompt used for unconditional classifier free guidance\n*  ```seed``` - By default it is randomly chosen, to specify seed change thise value.\n*  ```optimization_step``` - number of optimization steps for each denoising step\n* ```optim_lr``` - learning rate\n* ```with_lr_decay```  - if True, overrides `optim_lr`, and the learning rate will decay during the optimization process in the range of `scale_range`\n\nAfter updating the config file, run the following command:\n```\npython run.py --config_path configs/guidance_config.yaml\n```\n\nOnce the method is done, the video will be saved to the ```output_path``` under `result.mp4`.\n\n\n# Tips\n* To get better samples from the T2V model, we used the prefix text ```\"Amazing quality, masterpiece, \"``` for inversion and edits.\n* If the video contains more complex motion/small objects, try increasing number of optimization steps - ```optimization_step: 30```.\n* For large deviation in structure between the source and target objects, try using a lower lr - ```scale_range:[0.005, 0.002]```,\n*  or adding the source object to the negative prompt text.\n\n# Measuring motion fidelity\nWe also provide the code for calculating the motion fidelity metric introduced in the paper (Section 5.1).\nTo calculate the motion fidelity metric, first follow the instructions [here](https://github.com/facebookresearch/co-tracker) to install Co-Tracker and download their checkpoint.\nThen, run the following command:\n```\npython motion_fidelity_score.py --config_path configs/motion_fidelity_config.yaml\n```\n\n\n\n# Citation\n```\n@article{yatim2023spacetime,\n        title = {Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer},\n        author = {Yatim, Danah and Fridman, Rafail and Bar-Tal, Omer and Kasten, Yoni and Dekel, Tali},\n        journal={arXiv preprint arxiv:2311.17009},\n        year={2023}\n        }\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdiffusion-motion-transfer%2Fdiffusion-motion-transfer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdiffusion-motion-transfer%2Fdiffusion-motion-transfer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdiffusion-motion-transfer%2Fdiffusion-motion-transfer/lists"}