{"id":15131521,"url":"https://github.com/chenyangqiqi/fatezero","last_synced_at":"2025-04-12T15:42:47.092Z","repository":{"id":143202316,"uuid":"614950419","full_name":"ChenyangQiQi/FateZero","owner":"ChenyangQiQi","description":"[ICCV 2023 Oral] \"FateZero: Fusing Attentions for Zero-shot Text-based Video Editing\"","archived":false,"fork":false,"pushed_at":"2023-08-14T00:25:26.000Z","size":209505,"stargazers_count":1142,"open_issues_count":12,"forks_count":107,"subscribers_count":13,"default_branch":"main","last_synced_at":"2025-04-03T16:12:41.664Z","etag":null,"topics":["image-editing","stable-diffusion","text-driven-editing","video-editing","video-style-transfer"],"latest_commit_sha":null,"homepage":"http://fate-zero-edit.github.io/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ChenyangQiQi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-03-16T16:35:52.000Z","updated_at":"2025-03-30T02:09:45.000Z","dependencies_parsed_at":null,"dependency_job_id":"e3a45110-39ff-4755-bbd6-b0d34c179cdb","html_url":"https://github.com/ChenyangQiQi/FateZero","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChenyangQiQi%2FFateZero","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChenyangQiQi%2FFateZero/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChenyangQiQi%2FFateZero/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChenyangQiQi%2FFateZero/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ChenyangQiQi","download_url":"https://codeload.github.com/ChenyangQiQi/FateZero/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248590929,"owners_count":21129916,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["image-editing","stable-diffusion","text-driven-editing","video-editing","video-style-transfer"],"created_at":"2024-09-26T03:42:53.131Z","updated_at":"2025-04-12T15:42:47.071Z","avatar_url":"https://github.com/ChenyangQiQi.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"### FateZero: Fusing Attentions for Zero-shot Text-based Video Editing (ICCV23 Oral')\n\n[Chenyang Qi](https://chenyangqiqi.github.io/), [Xiaodong Cun](http://vinthony.github.io/), [Yong Zhang](https://yzhang2016.github.io), [Chenyang Lei](https://chenyanglei.github.io/), [Xintao Wang](https://xinntao.github.io/), [Ying Shan](https://scholar.google.com/citations?hl=zh-CN\u0026user=4oXBp9UAAAAJ), and [Qifeng Chen](https://cqf.io)\n\n\u003ca href='https://arxiv.org/abs/2303.09535'\u003e\u003cimg src='https://img.shields.io/badge/ArXiv-2303.09535-red'\u003e\u003c/a\u003e \n\u003ca href='https://fate-zero-edit.github.io/'\u003e\u003cimg src='https://img.shields.io/badge/Project-Page-Green'\u003e\u003c/a\u003e  [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ChenyangQiQi/FateZero/blob/main/colab_fatezero.ipynb)\n[![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/chenyangqi/FateZero)\n[![GitHub](https://img.shields.io/github/stars/ChenyangQiQi/FateZero?style=social)](https://github.com/ChenyangQiQi/FateZero)\n\n\n\u003ctable class=\"center\"\u003e\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/car_posche_local_blend.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/3_sunflower_vangogh_conat_result.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/attri/16_sq_eat_02_concat_result.gif\"\u003e\u003c/td\u003e\n  \u003ctr\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"silver jeep ➜ posche car\"\u003c/td\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"+ Van Gogh style\"\u003c/td\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"squirrel,Carrot ➜ rabbit,eggplant\"\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table \u003e\n\n## 🎏 Abstract\n\u003cb\u003eTL; DR: \u003cfont color=\"red\"\u003eFateZero\u003c/font\u003e is the first zero-shot framework for text-driven video editing via pretrained diffusion models without training.\u003c/b\u003e\n\n\u003cdetails\u003e\u003csummary\u003eCLICK for the full abstract\u003c/summary\u003e\n\n\n\u003e The diffusion-based generative models have achieved\nremarkable success in text-based image generation. However,\nsince it contains enormous randomness in generation\nprogress, it is still challenging to apply such models for\nreal-world visual content editing, especially in videos. In\nthis paper, we propose \u003cfont color=\"red\"\u003eFateZero\u003c/font\u003e, a zero-shot text-based editing method on real-world videos without per-prompt\ntraining or use-specific mask. To edit videos consistently,\nwe propose several techniques based on the pre-trained\nmodels. Firstly, in contrast to the straightforward DDIM\ninversion technique, our approach captures intermediate\nattention maps during inversion, which effectively retain\nboth structural and motion information. These maps are\ndirectly fused in the editing process rather than generated\nduring denoising. To further minimize semantic leakage of\nthe source video, we then fuse self-attentions with a blending\nmask obtained by cross-attention features from the source\nprompt. Furthermore, we have implemented a reform of the\nself-attention mechanism in denoising UNet by introducing\nspatial-temporal attention to ensure frame consistency. Yet\nsuccinct, our method is the first one to show the ability of\nzero-shot text-driven video style and local attribute editing\nfrom the trained text-to-image model. We also have a better\nzero-shot shape-aware editing ability based on the text-to-video\nmodel. Extensive experiments demonstrate our\nsuperior temporal consistency and editing capability than\nprevious works.\n\u003c/details\u003e\n\n## 📋 Changelog\n- 2023.06.10 Two examples of large motion and multiple object.\n- 2023.04.18 Code refactoring and support local blending using [blend_latents option](config/teaser/jeep_posche_local_latent_blend.yaml). \n\u003c!-- - 2023.04.18 Code refactoring. Release a config with optional local blending [here](config/teaser/jeep_posche_local_latent_blend.yaml) using `blend_latents` option.  --\u003e\n- 2023.04.04 Release Enhanced Tuning-a-Video [configs](config/tune) and shape editing [ckpts](https://huggingface.co/chenyangqi/), [data](https://github.com/ChenyangQiQi/FateZero/releases/download/v0.0.1/shape.zip) and [config](config/shape)\n- 2023.03.31 Refine hugging face demo.\n\u003c!-- - 2023.03.27 Excited to Release [`Hugging face demo`](https://huggingface.co/spaces/chenyangqi/FateZero)! (refinement is in progress) Enjoy the fun of zero-shot video editing freely!\n- 2023.03.27 Release [`attribute editing config`](config/attribute) and \n  [`data`](https://github.com/ChenyangQiQi/FateZero/releases/download/v0.0.1/attribute.zip) used in the paper. --\u003e\n- 2023.03.27 Excited to Release [`Hugging face demo`](https://huggingface.co/spaces/chenyangqi/FateZero), [`attribute editing config`](config/attribute) and [`data`](https://github.com/ChenyangQiQi/FateZero/releases/download/v0.0.1/attribute.zip)\n \u003c!-- used in the paper. Enjoy the fun of zero-shot video editing freely! --\u003e\n\u003c!-- - 2023.03.27 Release [`attribute editing config`](config/attribute) and  --\u003e\n  \u003c!-- [`data`](https://github.com/ChenyangQiQi/FateZero/releases/download/v0.0.1/attribute.zip) used in the paper. --\u003e\n\n\n\u003c!-- - 2023.03.22 Upload a `colab notebook` [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ChenyangQiQi/FateZero/blob/main/colab_fatezero.ipynb).\n- 2023.03.22 Release [`style editing config`](config/style) and \n  \u003c!--[`data`](https://hkustconnect-my.sharepoint.com/:u:/g/personal/cqiaa_connect_ust_hk/EaTqRAuW0eJLj0z_JJrURkcBZCC3Zvgsdo6zsXHhpyHhHQ?e=FzuiNG) --\u003e\n  \u003c!-- [`data`](https://github.com/ChenyangQiQi/FateZero/releases/download/v0.0.1/style.zip)\n  used in the paper. --\u003e\n\n- 2023.03.22 Upload [`style editing config`](config/style) and \u003c!--[`data`](https://hkustconnect-my.sharepoint.com/:u:/g/personal/cqiaa_connect_ust_hk/EaTqRAuW0eJLj0z_JJrURkcBZCC3Zvgsdo6zsXHhpyHhHQ?e=FzuiNG) --\u003e\n  [`data`](https://github.com/ChenyangQiQi/FateZero/releases/download/v0.0.1/style.zip)\n   and a `colab notebook` [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ChenyangQiQi/FateZero/blob/main/colab_fatezero.ipynb).\n\n- 2023.03.21 Provide [Editing guidance](docs/EditingGuidance.md) for in-the-wild video. Update the [`config`](config/low_resource_teaser) for lower resources computers (16G GPU and less than 16G CPU RAM).\n\u003c!-- - in `config/low_resource_teaser`. --\u003e\n\u003c!-- -  Welcome to play and give feedback!  --\u003e\n\u003c!-- A new option store all the attentions in hard disk, which require less ram. --\u003e\n- 2023.03.17 Release Code and Paper!\n\n## 🚧 Todo\n\n\u003cdetails\u003e\u003csummary\u003eClick for Previous todos \u003c/summary\u003e\n\n- [x] Release the edit config and data for all results, Tune-a-video optimization\n- [x] Memory and runtime profiling and Editing guidance documents\n- [x] Colab and hugging-face\n- [x] code refactoring\n\u003c/details\u003e\n\n- [ ] time \u0026 memory optimization\n- [ ] Release more application\n\n## 🛡 Setup Environment\nOur method is tested using cuda11, fp16 of accelerator and xformers on a single A100 or 3090.\n\n```bash\nconda create -n fatezero38 python=3.8\nconda activate fatezero38\n\npip install -r requirements.txt\n```\n\n`xformers` is recommended for A100 GPU to save memory and running time. \n\n\u003cdetails\u003e\u003csummary\u003eClick for xformers installation \u003c/summary\u003e\n\nWe find its installation not stable. You may try the following wheel:\n```bash\nwget https://github.com/ShivamShrirao/xformers-wheels/releases/download/4c06c79/xformers-0.0.15.dev0+4c06c79.d20221201-cp38-cp38-linux_x86_64.whl\npip install xformers-0.0.15.dev0+4c06c79.d20221201-cp38-cp38-linux_x86_64.whl\n```\n\n\u003c/details\u003e\n\nValidate the installation by \n```\npython test_install.py\n```\nYou may download all data and checkpoints using the following bash command\n```\nbash download_all.sh\n```\nThe above command take minutes and 100GB. Or you may download the required data and ckpts latter according to your interests.\n\nOur environment is similar to Tune-A-video ([official](https://github.com/showlab/Tune-A-Video), [unofficial](https://github.com/bryandlee/Tune-A-Video))  and [prompt-to-prompt](https://github.com/google/prompt-to-prompt/). You may check them for more details.\n\n\n## ⚔️ FateZero Editing\n\n#### Style and Attribute Editing in Teaser\n\nDownload the [stable diffusion v1-4](https://huggingface.co/CompVis/stable-diffusion-v1-4) (or other interesting image diffusion model) and put it to `./ckpt/stable-diffusion-v1-4`. \n\n\u003cdetails\u003e\u003csummary\u003eClick for the bash command: \u003c/summary\u003e\n \n```\nmkdir ./ckpt\ncd ./ckpt\n# download from huggingface face, takes 20G space\ngit lfs install\ngit clone https://huggingface.co/CompVis/stable-diffusion-v1-4\n```\n\u003c/details\u003e\n\nThen, you could reproduce style and shape editing results in our teaser by running:\n\n```bash\naccelerate launch test_fatezero.py --config config/teaser/jeep_watercolor.yaml\n# or CUDA_VISIBLE_DEVICES=0 python test_fatezero.py --config config/teaser/jeep_watercolor.yaml\n```\n\n\u003cdetails\u003e\u003csummary\u003eThe result is saved at `./result` . (Click for directory structure) \u003c/summary\u003e\n\n```\nresult\n├── teaser\n│   ├── jeep_posche\n│   ├── jeep_watercolor\n│           ├── cross-attention  # visualization of cross-attention during inversion\n│           ├── sample           # result\n│           ├── train_samples    # the input video\n\n```\n\n\u003c/details\u003e\n\nEditing 8 frames on an Nvidia 3090, use `100G CPU memory, 12G GPU memory` for editing. We also provide some [`low-cost setting`](config/low_resource_teaser) of style editing by different hyper-parameters on a 16GB GPU. \nYou may try these low-cost settings on colab.\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ChenyangQiQi/FateZero/blob/main/colab_fatezero.ipynb)\n\nMore speed and hardware benchmarks are [here](docs/EditingGuidance.md#ddim-hyperparameters).\n\n#### Shape and large motion editing with Tune-A-Video\n\nBesides style and attribution editing above, we also provide a `Tune-A-Video` checkpoint. You may download from [onedrive](https://hkustconnect-my.sharepoint.com/:f:/g/personal/cqiaa_connect_ust_hk/EviSTWoAOs1EmHtqZruq50kBZu1E8gxDknCPigSvsS96uQ?e=492khj) or from [hugging face model repository](https://huggingface.co/chenyangqi/jeep_tuned_200). Then move it to `./ckpt/jeep_tuned_200/`.\n\n\n\u003cdetails\u003e\u003csummary\u003eClick for the bash command: \u003c/summary\u003e\n\n```\nmkdir ./ckpt\ncd ./ckpt\n# download from huggingface face, takes 10G space\ngit lfs install\ngit clone https://huggingface.co/chenyangqi/jeep_tuned_200\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\u003csummary\u003eThe directory structure should be like this: (Click for directory structure) \u003c/summary\u003e\n\n```\nckpt\n├── stable-diffusion-v1-4\n├── jeep_tuned_200\n...\ndata\n├── car-turn\n│   ├── 00000000.png\n│   ├── 00000001.png\n│   ├── ...\nvideo_diffusion\n```\n\u003c/details\u003e\n\nYou could reproduce the shape editing result in our teaser by running:\n\n```bash\naccelerate launch test_fatezero.py --config config/teaser/jeep_posche.yaml\n```\n\n\n### Reproduce other results in the paper\n\u003c!-- Download the data of [style editing](https://hkustconnect-my.sharepoint.com/:u:/g/personal/cqiaa_connect_ust_hk/EaTqRAuW0eJLj0z_JJrURkcBZCC3Zvgsdo6zsXHhpyHhHQ?e=FzuiNG) and [attribute editing](https://hkustconnect-my.sharepoint.com/:u:/g/personal/cqiaa_connect_ust_hk/Ee7J2IzZuaVGkefh-ZRp1GwB7RCUYU7MVJCKqeNWmOIpfg?e=dcOwb7)\n--\u003e\nDownload the data\nfrom [onedrive](https://hkustconnect-my.sharepoint.com/:f:/g/personal/cqiaa_connect_ust_hk/EkIeHj3CQiBNhm6iEEhJQZwBEBJNCGt3FsANmyqeAYbuXQ?e=SCPJlu) or from Github [Release](https://github.com/ChenyangQiQi/FateZero/releases/tag/v0.0.1).\n\u003cdetails\u003e\u003csummary\u003eClick for wget bash command: \u003c/summary\u003e\n \n```\nwget https://github.com/ChenyangQiQi/FateZero/releases/download/v0.0.1/attribute.zip\nwget https://github.com/ChenyangQiQi/FateZero/releases/download/v0.0.1/style.zip\nwget https://github.com/ChenyangQiQi/FateZero/releases/download/v0.0.1/shape.zip\n```\n\u003c/details\u003e\n\nUnzip and Place it in ['./data'](data). Then use the commands in ['config/style'](config/style) and ['config/attribute'](config/attribute) to get the results.\n\nTo reproduce other shape editing results, download Tune-A-Video checkpoints from [huggingface](https://huggingface.co/chenyangqi/) :\n\n\u003cdetails\u003e\u003csummary\u003eClick for the bash command: \u003c/summary\u003e\n\n```\nmkdir ./ckpt\ncd ./ckpt\n# download from huggingface face, takes 10G space\ngit lfs install\ngit clone https://huggingface.co/chenyangqi/man_skate_250\ngit clone https://huggingface.co/chenyangqi/swan_150\n```\n\u003c/details\u003e\n\nThen use the commands in ['config/shape'](config/shape).\n\nFor above Tune-A-Video checkpoints, we fintune stable diffusion with a synthetic negative-prompt [dataset](https://github.com/ChenyangQiQi/FateZero/releases/download/v0.0.1/negative_reg.zip) for regularization and low-rank conovlution for temporal-consistent generation using [tuning config](./config/tune/)\n\n\u003cdetails\u003e\u003csummary\u003eClick for the bash command example: \u003c/summary\u003e\n\n```\ncd ./data\nwget https://github.com/ChenyangQiQi/FateZero/releases/download/v0.0.1/negative_reg.zip\nunzip negative_reg\ncd ..\naccelerate launch train_tune_a_video.py --config config/tune/jeep.yaml\n```\nTo evaluate our results quantitatively, we provide `CLIP/frame_acc_tem_con.py` to calculate frame accuracy and temporal consistency using pretrained CLIP.\n\u003c/details\u003e\n\n## Editing guidance for YOUR video\nWe provided a editing guidance for in-the-wild video [here](./docs/EditingGuidance.md). The work is still in progress. Welcome to give your feedback in issues.\n\n## Style Editing Results with Stable Diffusion\nWe show the difference between the source prompt and the target prompt in the box below each video.\n\nNote mp4 and gif files in this GitHub page are compressed. \nPlease check our [Project Page](https://fate-zero-edit.github.io/) for mp4 files of original video editing results.\n\u003ctable class=\"center\"\u003e\n\n\u003ctr\u003e\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/style/1_surf_ukiyo_01_concat_result.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/style/2_car_watercolor_01_concat_result.gif\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"docs/gif_results/style/6_lily_monet_01_concat_result.gif\"\u003e\u003c/td\u003e\n  \u003c!-- \u003ctd\u003e\u003cimg src=\"https://tuneavideo.github.io/assets/results/tuneavideo/man-skiing/wonder-woman.gif\"\u003e\u003c/td\u003e              \n  \u003ctd\u003e\u003cimg src=\"https://tuneavideo.github.io/assets/results/tuneavideo/man-skiing/pink-sunset.gif\"\u003e\u003c/td\u003e --\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"+ Ukiyo-e style\"\u003c/td\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"+ watercolor painting\"\u003c/td\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"+ Monet style\"\u003c/td\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/style/4_rabit_pokemon_01_concat_result.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/style/5_train_shikai_01_concat_result.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/style/parkour_watercolor_0_-1.gif\"\u003e\u003c/td\u003e\n\n\u003c/tr\u003e\n\u003ctr\u003e\n\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"+ Pokémon cartoon style\"\u003c/td\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"+ Makoto Shinkai style\"\u003c/td\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"+ watercolor painting\"\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n## Attribute Editing Results with Stable Diffusion\n\u003ctable class=\"center\"\u003e\n\n\u003ctr\u003e\n\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/attri/15_rabbit_eat_01_concat_result.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/attri/15_rabbit_eat_02_concat_result.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/attri/15_rabbit_eat_04_concat_result.gif\"\u003e\u003c/td\u003e\n\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"rabbit, strawberry ➜ white rabbit, flower\"\u003c/td\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"rabbit, strawberry ➜ squirrel, carrot\"\u003c/td\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"rabbit, strawberry ➜ white rabbit, leaves\"\u003c/td\u003e\n\n\u003c/tr\u003e\n\n\n\u003ctr\u003e\n\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/attri/13_bear_tiger_leopard_lion_01_concat_result.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/attri/13_bear_tiger_leopard_lion_02_concat_result.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/attri/13_bear_tiger_leopard_lion_03_concat_result.gif\"\u003e\u003c/td\u003e\n\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"bear ➜ a red tiger\"\u003c/td\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"bear ➜ a yellow leopard\"\u003c/td\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"bear ➜ a brown lion\"\u003c/td\u003e\n\n\u003c/tr\u003e\n\u003ctr\u003e\n\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/attri/14_cat_grass_tiger_corgin_02_concat_result.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/attri/14_cat_grass_tiger_corgin_03_concat_result.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/attri/14_cat_grass_tiger_corgin_04_concat_result.gif\"\u003e\u003c/td\u003e\n\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"cat ➜ black cat, grass...\"\u003c/td\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"cat ➜ red tiger\"\u003c/td\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"cat ➜ Shiba-Inu\"\u003c/td\u003e\n\n\u003c/tr\u003e\n\n\n\u003ctr\u003e\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/attri/goldfish_yellow.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/attri/16_sq_eat_04_concat_result.gif\"\u003e\u003c/td\u003e\n\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/attri/16_sq_eat_03_concat_result.gif\"\u003e\u003c/td\u003e\n\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"orange fish ➜ yellow fish\"\u003c/td\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"squirrel ➜ robot squirrel\"\u003c/td\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"squirrel, Carrot ➜ robot mouse, screwdriver\"\u003c/td\u003e\n\n\u003c/tr\u003e\n\n\u003ctr\u003e\n\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/attri/10_bus_gpu_01_concat_result.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/attri/11_dog_robotic_corgin_01_concat_result.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/attri/11_dog_robotic_corgin_02_concat_result.gif\"\u003e\u003c/td\u003e\n\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"bus ➜ GPU\"\u003c/td\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"gray dog ➜ yellow corgi\"\u003c/td\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"gray dog ➜ robotic dog\"\u003c/td\u003e\n\n\u003c/tr\u003e\n\u003ctr\u003e\n\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/attri/9_duck_rubber_01_concat_result.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/attri/12_fox_snow_wolf_01_concat_result.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/attri/12_fox_snow_wolf_02_concat_result.gif\"\u003e\u003c/td\u003e\n\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"white duck ➜ yellow rubber duck\"\u003c/td\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"grass ➜ snow\"\u003c/td\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"white fox ➜ grey wolf\"\u003c/td\u003e\n\n\u003c/tr\u003e\n\n\n\u003c/table\u003e\n\n## Shape and large motion editing with Tune-A-Video\n\u003ctable class=\"center\"\u003e\n\n\u003ctr\u003e\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/shape/17_car_posche_01_concat_result.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/shape/18_swan_01_concat_result.gif\"\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cimg src=\"docs/gif_results/shape/18_swan_02_concat_result.gif\"\u003e\u003c/td\u003e\n  \u003c!-- \u003ctd\u003e\u003cimg src=\"https://tuneavideo.github.io/assets/results/tuneavideo/man-skiing/wonder-woman.gif\"\u003e\u003c/td\u003e              \n  \u003ctd\u003e\u003cimg src=\"https://tuneavideo.github.io/assets/results/tuneavideo/man-skiing/pink-sunset.gif\"\u003e\u003c/td\u003e --\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"silver jeep ➜ posche car\"\u003c/td\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"Swan ➜ White Duck\"\u003c/td\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"Swan ➜ Pink flamingo\"\u003c/td\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/shape/19_man_wonder_01_concat_result.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/shape/19_man_wonder_02_concat_result.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"docs/gif_results/shape/19_man_wonder_03_concat_result.gif\"\u003e\u003c/td\u003e\n\n\u003c/tr\u003e\n\u003ctr\u003e\n\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"A man ➜ A Batman\"\u003c/td\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"A man ➜ A Wonder Woman, With cowboy hat\"\u003c/td\u003e\n  \u003ctd width=25% style=\"text-align:center;\"\u003e\"A man ➜ A Spider-Man\"\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n## 🕹 Online Demo\nThanks to AK and the team from Hugging Face for providing computing resources to support our Hugging-face Demo, which supports up to 30 steps DDIM steps.\n[![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/chenyangqi/FateZero).\n\nYou may use the UI for testing FateZero built with gradio locally.\n```\ngit clone https://huggingface.co/spaces/chenyangqi/FateZero\npython app_fatezero.py\n# we will merge the FateZero on hugging face with that in github repo latter\n```\n\nWe also provide a Colab demo, which supports 10 DDIM steps.\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ChenyangQiQi/FateZero/blob/main/colab_fatezero.ipynb)\nYou may launch the colab as a jupyter notebook on your local machine.\nWe will refine and optimize the above demos in the following days.\n\n## 📀 Demo Video\n\nhttps://user-images.githubusercontent.com/45789244/225698509-79c14793-3153-4bba-9d6e-ede7d811d7f8.mp4\n\nThe video here is compressed due to the size limit of GitHub.\nThe original full-resolution video is [here](https://hkustconnect-my.sharepoint.com/:v:/g/personal/cqiaa_connect_ust_hk/EXKDI_nahEhKtiYPvvyU9SkBDTG2W4G1AZ_vkC7ekh3ENw?e=Xhgtmk).\n\n\n## 📍 Citation \n\n```\n@article{qi2023fatezero,\n      title={FateZero: Fusing Attentions for Zero-shot Text-based Video Editing}, \n      author={Chenyang Qi and Xiaodong Cun and Yong Zhang and Chenyang Lei and Xintao Wang and Ying Shan and Qifeng Chen},\n      year={2023},\n      journal={arXiv:2303.09535},\n}\n``` \n\n\n## 💗 Acknowledgements\n\nThis repository borrows heavily from [Tune-A-Video](https://github.com/showlab/Tune-A-Video) and [prompt-to-prompt](https://github.com/google/prompt-to-prompt/). Thanks to the authors for sharing their code and models.\n\n## 🧿 Maintenance\n\nThis is the codebase for our research work. We are still working hard to update this repo, and more details are coming in days. If you have any questions or ideas to discuss, feel free to contact [Chenyang Qi](cqiaa@connect.ust.hk) or [Xiaodong Cun](vinthony@gmail.com).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchenyangqiqi%2Ffatezero","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchenyangqiqi%2Ffatezero","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchenyangqiqi%2Ffatezero/lists"}