{"id":13488888,"url":"https://github.com/williamyang1991/FRESCO","last_synced_at":"2025-03-28T02:31:22.435Z","repository":{"id":227273015,"uuid":"770954805","full_name":"williamyang1991/FRESCO","owner":"williamyang1991","description":"[CVPR 2024] FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation","archived":false,"fork":false,"pushed_at":"2024-03-31T12:32:43.000Z","size":10421,"stargazers_count":541,"open_issues_count":18,"forks_count":51,"subscribers_count":11,"default_branch":"main","last_synced_at":"2024-03-31T13:34:56.568Z","etag":null,"topics":["controlnet","diffusion","video-processing"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/williamyang1991.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-03-12T12:41:24.000Z","updated_at":"2024-04-15T09:49:34.138Z","dependencies_parsed_at":"2024-04-15T09:59:40.889Z","dependency_job_id":null,"html_url":"https://github.com/williamyang1991/FRESCO","commit_stats":null,"previous_names":["williamyang1991/fresco"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/williamyang1991%2FFRESCO","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/williamyang1991%2FFRESCO/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/williamyang1991%2FFRESCO/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/williamyang1991%2FFRESCO/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/williamyang1991","download_url":"https://codeload.github.com/williamyang1991/FRESCO/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245957671,"owners_count":20700314,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["controlnet","diffusion","video-processing"],"created_at":"2024-07-31T18:01:23.591Z","updated_at":"2025-03-28T02:31:18.616Z","avatar_url":"https://github.com/williamyang1991.png","language":"Jupyter Notebook","funding_links":[],"categories":["Video Generation","Jupyter Notebook"],"sub_categories":[],"readme":"# FRESCO - Official PyTorch Implementation\n\n\n**FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation**\u003cbr\u003e\n[Shuai Yang](https://williamyang1991.github.io/), [Yifan Zhou](https://zhouyifan.net/), [Ziwei Liu](https://liuziwei7.github.io/) and [Chen Change Loy](https://www.mmlab-ntu.com/person/ccloy/)\u003cbr\u003e\nin CVPR 2024 \u003cbr\u003e\n[**Project Page**](https://www.mmlab-ntu.com/project/fresco/) | [**Paper**](https://arxiv.org/abs/2403.12962) | [**Supplementary Video**](https://youtu.be/jLnGx5H-wLw) | [**Input Data and Video Results**](https://drive.google.com/file/d/12BFx3hp8_jp9m0EmKpw-cus2SABPQx2Q/view?usp=sharing) \u003cbr\u003e\n\n\u003ca href=\"https://huggingface.co/spaces/PKUWilliamYang/FRESCO\"\u003e\u003cimg src=\"https://huggingface.co/datasets/huggingface/badges/raw/main/open-in-hf-spaces-sm-dark.svg\" alt=\"Web Demo\"\u003e\u003c/a\u003e \n\n**Abstract:** *The remarkable efficacy of text-to-image diffusion models has motivated extensive exploration of their potential application in video domains.\nZero-shot methods seek to extend image diffusion models to videos without necessitating model training.\nRecent methods mainly focus on incorporating inter-frame correspondence into attention mechanisms. However, the soft constraint imposed on determining where to attend to valid features can sometimes be insufficient, resulting in temporal inconsistency.\nIn this paper, we introduce FRESCO, intra-frame correspondence alongside inter-frame correspondence to establish a more robust spatial-temporal constraint. This enhancement ensures a more consistent transformation of semantically similar content across frames. Beyond mere attention guidance, our approach involves an explicit update of features to achieve high spatial-temporal consistency with the input video, significantly improving the visual coherence of the resulting translated videos.\nExtensive experiments demonstrate the effectiveness of our proposed framework in producing high-quality, coherent videos, marking a notable improvement over existing zero-shot methods.*\n\n**Features**:\u003cbr\u003e\n- **Temporal consistency**: use intra-and inter-frame constraint with better consistency and coverage than optical flow alone.\n    - Compared with our previous work [Rerender-A-Video](https://github.com/williamyang1991/Rerender_A_Video), FRESCO is more robust to large and quick motion.\n- **Zero-shot**: no training or fine-tuning required.\n- **Flexibility**: compatible with off-the-shelf models (e.g., [ControlNet](https://github.com/lllyasviel/ControlNet), [LoRA](https://civitai.com/)) for customized translation.\n\nhttps://github.com/williamyang1991/FRESCO/assets/18130694/aad358af-4d27-4f18-b069-89a1abd94d38\n\n\n## Updates\n- [05/2024] The Diffusers pipeline is available: [FRESCO Community Pipeline](https://github.com/huggingface/diffusers/tree/main/examples/community#fresco).\n- [04/2024] Integrated to 🤗 [Hugging Face](https://huggingface.co/spaces/PKUWilliamYang/FRESCO). Enjoy the web demo!\n- [03/2024] Paper is released.\n- [03/2024] Code is released.\n- [03/2024] This website is created.\n\n### TODO\n- [x] ~~Integrate into Diffusers~~\n- [x] ~~Add Huggingface web demo~~\n- [x] ~~Add webUI.~~\n- [x] ~~Update readme~~\n- [x] ~~Upload paper to arXiv, release related material~~\n\n## Installation\n\n1. Clone the repository. \n\n```shell\ngit clone https://github.com/williamyang1991/FRESCO.git\ncd FRESCO\n```\n\n2. You can simply set up the environment with pip based on [requirements.txt](https://github.com/williamyang1991/FRESCO/blob/main/requirements.txt)\n    - Create a conda environment and install torch \u003e= 2.0.0. Here is an example script to install torch 2.0.0 + CUDA 11.8 :\n    ```\n    conda create --name diffusers python==3.8.5\n    conda activate diffusers\n    pip install torch==2.0.0 torchvision==0.15.1 --index-url https://download.pytorch.org/whl/cu118\n    ```\n    - Run `pip install -r requirements.txt` in an environment where torch is installed.\n    - We have tested on torch 2.0.0/2.1.0 and diffusers 0.19.3\n    - If you use new versions of diffusers, you need to modify [my_forward()](https://github.com/williamyang1991/FRESCO/blob/fb991262615665de88f7a8f2cc903d9539e1b234/src/diffusion_hacked.py#L496)\n\n3. Run the installation script. The required models will be downloaded in `./model`, `./src/ControlNet/annotator` and `./src/ebsynth/deps/ebsynth/bin`.\n    - Requires access to huggingface.co\n\n```shell\npython install.py\n```\n\n4. You can run the demo with `run_fresco.py`\n\n```shell\npython run_fresco.py ./config/config_music.yaml\n```\n\n5. For issues with Ebsynth, please refer to [issues](https://github.com/williamyang1991/Rerender_A_Video#issues)\n\n\n## (1) Inference\n\n### WebUI (recommended)\n\n```\npython webUI.py\n```\nThe Gradio app also allows you to flexibly change the inference options. Just try it for more details. \n\nUpload your video, input the prompt, select the model and seed, and hit:\n- **Run Key Frames**: detect keyframes, translate all keyframes.\n- **Run Propagation**: propagate the keyframes to other frames for full video translation\n- **Run All**: **Run Key Frames** and **Run Propagation**\n\nSelect the model:\n- **Base model**: base Stable Diffusion model (SD 1.5)\n    - Stable Diffusion 1.5: official model\n    - [rev-Animated](https://huggingface.co/stablediffusionapi/rev-animated): a semi-realistic (2.5D) model\n    - [realistic-Vision](https://huggingface.co/SG161222/Realistic_Vision_V2.0): a photo-realistic model\n    - [flat2d-animerge](https://huggingface.co/stablediffusionapi/flat-2d-animerge): a cartoon model\n    - You can add other models on huggingface.co by modifying this [line](https://github.com/williamyang1991/FRESCO/blob/1afcca9c7b1bc1ac68254f900be9bd768fbb6988/webUI.py#L362) \n   \n![overview](https://github.com/williamyang1991/FRESCO/assets/18130694/6ce5d54e-b020-4e43-95e7-72ab1783f482)\n\nWe provide abundant advanced options to play with\n\n\u003c/details\u003e\n\n\u003cdetails id=\"option1\"\u003e\n\u003csummary\u003e \u003cb\u003eAdvanced options for single frame processing\u003c/b\u003e\u003c/summary\u003e\n\n1. **Frame resolution**: resize the short side of the video to 512.\n2. ControlNet related:\n   - **ControlNet strength**: how well the output matches the input control edges\n   - **Control type**: HED edge, Canny edge, Depth map\n   - **Canny low/high threshold**: low values for more edge details\n3. SDEdit related:\n   - **Denoising strength**: repaint degree (low value to make the output look more like the original video)\n   - **Preserve color**: preserve the color of the original video\n4. SD related:\n   - **Steps**: denoising step\n   - **CFG scale**: how well the output matches the prompt\n   - **Added prompt/Negative prompt**: supplementary prompts\n5. FreeU related:\n   - **FreeU first/second-stage backbone factor**: =1 do nothing; \u003e1 enhance output color and details\n   - **FreeU first/second-stage skip factor**: =1 do nothing; \u003c1 enhance output color and details\n\n\u003c/details\u003e\n\n\u003cdetails id=\"option2\"\u003e\n\u003csummary\u003e \u003cb\u003eAdvanced options for FRESCO constraints\u003c/b\u003e\u003c/summary\u003e\n\n1. Keyframe related\n   - **Number of frames**: Total frames to be translated\n   - **Number of frames in a batch**: To avoid out-of-memory, use small batch size\n   - **Min keyframe interval (s_min)**: The keyframes will be detected at least every s_min frames\n   - **Max keyframe interval (s_max)**: The keyframes will be detected at most every s_max frames\n2. FRESCO constraints\n   - FRESCO-guided Attention:\n     - **spatial-guided attention**: Check to enable spatial-guided attention\n     - **cross-frame attention**: Check to enable efficient cross-frame attention\n     - **temporal-guided attention**: Check to enable temporal-guided attention\n   - FRESCO-guided optimization:\n     - **spatial-guided optimization**: Check to enable spatial-guided optimization\n     - **temporal-guided optimization**: Check to enable temporal-guided optimization\n3. **Background smoothing**: Check to enable background smoothing (best for static background)\n   \n\u003c/details\u003e\n\n\u003cdetails id=\"option3\"\u003e\n\u003csummary\u003e \u003cb\u003eAdvanced options for the full video translation\u003c/b\u003e\u003c/summary\u003e\n\n1. **Gradient blending**: apply Poisson Blending to reduce ghosting artifacts. May slow the process and increase flickers.\n2. **Number of parallel processes**: multiprocessing to speed up the process. Large value (4) is recommended.\n\u003c/details\u003e\n\n![option](https://github.com/williamyang1991/FRESCO/assets/18130694/72600758-1dff-4b7c-8f3f-65ee3909f8f6)\n\n### Command Line\n\nWe provide a flexible script `run_fresco.py` to run our method.\n\nSet the options via a config file. For example,\n```shell\npython run_fresco.py ./config/config_music.yaml\n```\nWe provide some examples of the config in `config` directory.\nMost options in the config is the same as those in WebUI.\nPlease check the explanations in the WebUI section.\n\nWe provide a separate Ebsynth python script `video_blend.py` with the temporal blending algorithm introduced in\n[Stylizing Video by Example](https://dcgi.fel.cvut.cz/home/sykorad/ebsynth.html) for interpolating style between key frames.\nIt can work on your own stylized key frames independently of our FRESCO algorithm.\n\n```python\nvideo_blend.py [-h] [--output OUTPUT] [--fps FPS] [--key_ind KEY_IND [KEY_IND ...]] [--key KEY] [--n_proc N_PROC] [-ps] [-ne] [-tmp] name\n\npositional arguments:\n  name                  Path to input video\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --output OUTPUT       Path to output video\n  --fps FPS             The FPS of output video\n  --key_ind KEY_IND [KEY_IND ...]\n                        key frame index\n  --key KEY             The subfolder name of stylized key frames\n  --n_proc N_PROC       The max process count\n  -ps                   Use poisson gradient blending\n  -ne                   Do not run ebsynth (use previous ebsynth output)\n  -tmp                  Keep temporary output\n```\nAn example\n```\npython video_blend.py ./output/dog/ --key keys --key_ind 0 11 23 33 49 60 72 82 93 106 120 137 151 170 182 193 213 228 238 252 262 288 299  --output ./output/dog/blend.mp4 --fps 24 --n_proc 4 -ps\n```\n\nFor the details, please refer to our previous work [Rerender-A-Video](https://github.com/williamyang1991/Rerender_A_Video/tree/main?tab=readme-ov-file#our-ebsynth-implementation) (The mainly difference is the way of specifying key frame index)\n\n## (2) Results\n\n### Key frame translation\n\n\u003ctable class=\"center\"\u003e\n\u003ctr\u003e\n  \u003ctd\u003e\u003cimg src=\"https://github.com/williamyang1991/FRESCO/assets/18130694/e8d5776a-37c5-49ae-8ab4-15669df6f572\" raw=true\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"https://github.com/williamyang1991/FRESCO/assets/18130694/8a792af6-555c-4e82-ac1e-5c2e1ee35fdb\" raw=true\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"https://github.com/williamyang1991/FRESCO/assets/18130694/10f9a964-85ac-4433-84c5-1611a6c2c434\" raw=true\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"https://github.com/williamyang1991/FRESCO/assets/18130694/0ec0fbf9-90dd-4d8b-964d-945b5f6687c2\" raw=true\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd width=26.5% align=\"center\"\u003ea red car turns in the winter\u003c/td\u003e\n  \u003ctd width=26.5% align=\"center\"\u003ean African American boxer wearing black boxing gloves punches towards the camera, cartoon style\u003c/td\u003e\n  \u003ctd width=26.5% align=\"center\"\u003ea cartoon spiderman in black suit, black shoes and white gloves is dancing\u003c/td\u003e\n  \u003ctd width=20.5% align=\"center\"\u003ea beautiful woman holding her glasses in CG style\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n\n### Full video translation\n\nhttps://github.com/williamyang1991/FRESCO/assets/18130694/bf8bfb82-5cb7-4b2f-8169-cf8dbf408b54\n\n## Citation\n\nIf you find this work useful for your research, please consider citing our paper:\n\n```bibtex\n@inproceedings{yang2024fresco,\n title = {FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation},\n author = {Yang, Shuai and Zhou, Yifan and Liu, Ziwei and and Loy, Chen Change},\n booktitle = {CVPR},\n year = {2024},\n}\n```\n\n## Acknowledgments\n\nThe code is mainly developed based on [Rerender-A-Video](https://github.com/williamyang1991/Rerender_A_Video), [ControlNet](https://github.com/lllyasviel/ControlNet), [Stable Diffusion](https://github.com/Stability-AI/stablediffusion), [GMFlow](https://github.com/haofeixu/gmflow) and [Ebsynth](https://github.com/jamriska/ebsynth).\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwilliamyang1991%2FFRESCO","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwilliamyang1991%2FFRESCO","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwilliamyang1991%2FFRESCO/lists"}