{"id":15108725,"url":"https://github.com/sczhou/propainter","last_synced_at":"2025-05-13T18:12:03.721Z","repository":{"id":192264847,"uuid":"685984512","full_name":"sczhou/ProPainter","owner":"sczhou","description":"[ICCV 2023] ProPainter: Improving Propagation and Transformer for Video Inpainting","archived":false,"fork":false,"pushed_at":"2025-02-19T12:07:56.000Z","size":71457,"stargazers_count":6052,"open_issues_count":65,"forks_count":693,"subscribers_count":59,"default_branch":"main","last_synced_at":"2025-04-25T17:51:17.440Z","etag":null,"topics":["object-removal","video-completion","video-inpainting","video-outpainting","watermark-removal"],"latest_commit_sha":null,"homepage":"https://shangchenzhou.com/projects/ProPainter/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sczhou.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-01T13:11:57.000Z","updated_at":"2025-04-25T07:55:18.000Z","dependencies_parsed_at":"2025-03-11T23:12:45.184Z","dependency_job_id":null,"html_url":"https://github.com/sczhou/ProPainter","commit_stats":null,"previous_names":["sczhou/propainter"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sczhou%2FProPainter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sczhou%2FProPainter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sczhou%2FProPainter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sczhou%2FProPainter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sczhou","download_url":"https://codeload.github.com/sczhou/ProPainter/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254000857,"owners_count":21997442,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["object-removal","video-completion","video-inpainting","video-outpainting","watermark-removal"],"created_at":"2024-09-25T22:23:22.477Z","updated_at":"2025-05-13T18:12:03.693Z","avatar_url":"https://github.com/sczhou.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n\u003cdiv class=\"logo\"\u003e\n   \u003ca href=\"https://shangchenzhou.com/projects/ProPainter/\"\u003e\n      \u003cimg src=\"assets/propainter_logo1_glow.png\" style=\"width: 180px\"\u003e\n   \u003c/a\u003e\n\u003c/div\u003e\n\n\u003ch1\u003eProPainter: Improving Propagation and Transformer for Video Inpainting\u003c/h1\u003e\n\n\u003cdiv\u003e\n    \u003ca href='https://shangchenzhou.com/' target='_blank'\u003eShangchen Zhou\u003c/a\u003e\u0026emsp;\n    \u003ca href='https://li-chongyi.github.io/' target='_blank'\u003eChongyi Li\u003c/a\u003e\u0026emsp;\n    \u003ca href='https://ckkelvinchan.github.io/' target='_blank'\u003eKelvin C.K. Chan\u003c/a\u003e\u0026emsp;\n    \u003ca href='https://www.mmlab-ntu.com/person/ccloy/' target='_blank'\u003eChen Change Loy\u003c/a\u003e\n\u003c/div\u003e\n\u003cdiv\u003e\n    S-Lab, Nanyang Technological University\u0026emsp; \n\u003c/div\u003e\n\n\u003cdiv\u003e\n    \u003cstrong\u003eICCV 2023\u003c/strong\u003e\n\u003c/div\u003e\n\n\u003cdiv\u003e\n    \u003ch4 align=\"center\"\u003e\n        \u003ca href=\"https://shangchenzhou.com/projects/ProPainter\" target='_blank'\u003e\n        \u003cimg src=\"https://img.shields.io/badge/🐳-Project%20Page-blue\"\u003e\n        \u003c/a\u003e\n        \u003ca href=\"https://arxiv.org/abs/2309.03897\" target='_blank'\u003e\n        \u003cimg src=\"https://img.shields.io/badge/arXiv-2309.03897-b31b1b.svg\"\u003e\n        \u003c/a\u003e\n        \u003ca href=\"https://youtu.be/92EHfgCO5-Q\" target='_blank'\u003e\n        \u003cimg src=\"https://img.shields.io/badge/Demo%20Video-%23FF0000.svg?logo=YouTube\u0026logoColor=white\"\u003e\n        \u003c/a\u003e\n        \u003ca href=\"https://huggingface.co/spaces/sczhou/ProPainter\" target='_blank'\u003e\n        \u003cimg src=\"https://img.shields.io/badge/Demo-%F0%9F%A4%97%20Hugging%20Face-blue\"\u003e\n        \u003c/a\u003e\n        \u003ca href=\"https://openxlab.org.cn/apps/detail/ShangchenZhou/ProPainter\" target='_blank'\u003e\n        \u003cimg src=\"https://img.shields.io/badge/Demo-%F0%9F%91%A8%E2%80%8D%F0%9F%8E%A8%20OpenXLab-blue\"\u003e\n        \u003c/a\u003e\n        \u003cimg src=\"https://api.infinitescript.com/badgen/count?name=sczhou/ProPainter\"\u003e\n    \u003c/h4\u003e\n\u003c/div\u003e\n\n⭐ If ProPainter is helpful to your projects, please help star this repo. Thanks! 🤗\n\n:open_book: For more visual results, go checkout our \u003ca href=\"https://shangchenzhou.com/projects/ProPainter/\" target=\"_blank\"\u003eproject page\u003c/a\u003e\n\n\n---\n\n\u003c/div\u003e\n\n\n## Update\n- **2023.11.09**: Integrated to :man_artist: [OpenXLab](https://openxlab.org.cn/apps). Try out online demo! [![OpenXLab](https://img.shields.io/badge/Demo-%F0%9F%91%A8%E2%80%8D%F0%9F%8E%A8%20OpenXLab-blue)](https://openxlab.org.cn/apps/detail/ShangchenZhou/ProPainter)\n- **2023.11.09**: Integrated to :hugs: [Hugging Face](https://huggingface.co/spaces). Try out online demo! [![Hugging Face](https://img.shields.io/badge/Demo-%F0%9F%A4%97%20Hugging%20Face-blue)](https://huggingface.co/spaces/sczhou/ProPainter)\n- **2023.09.24**: We remove the watermark removal demos officially to prevent the misuse of our work for unethical purposes.\n- **2023.09.21**: Add features for memory-efficient inference. Check our [GPU memory](https://github.com/sczhou/ProPainter#-memory-efficient-inference) requirements. 🚀\n- **2023.09.07**: Our code and model are publicly available. 🐳\n- **2023.09.01**: This repo is created.\n\n\n### TODO\n- [ ] Make a Colab demo.\n- [x] ~~Make a interactive Gradio demo.~~\n- [x] ~~Update features for memory-efficient inference.~~\n  \n## Results\n\n#### 👨🏻‍🎨 Object Removal\n\u003ctable\u003e\n\u003ctr\u003e\n   \u003ctd\u003e \n      \u003cimg src=\"assets/object_removal1.gif\"\u003e\n   \u003c/td\u003e\n   \u003ctd\u003e \n      \u003cimg src=\"assets/object_removal2.gif\"\u003e\n   \u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n#### 🎨 Video Completion\n\u003ctable\u003e\n\u003ctr\u003e\n   \u003ctd\u003e \n      \u003cimg src=\"assets/video_completion1.gif\"\u003e\n   \u003c/td\u003e\n   \u003ctd\u003e \n      \u003cimg src=\"assets/video_completion2.gif\"\u003e\n   \u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n   \u003ctd\u003e \n      \u003cimg src=\"assets/video_completion3.gif\"\u003e\n   \u003c/td\u003e\n   \u003ctd\u003e \n      \u003cimg src=\"assets/video_completion4.gif\"\u003e\n   \u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n\n\n## Overview\n![overall_structure](assets/ProPainter_pipeline.png)\n\n\n## Dependencies and Installation\n\n1. Clone Repo\n\n   ```bash\n   git clone https://github.com/sczhou/ProPainter.git\n   ```\n\n2. Create Conda Environment and Install Dependencies\n\n   ```bash\n   # create new anaconda env\n   conda create -n propainter python=3.8 -y\n   conda activate propainter\n\n   # install python dependencies\n   pip3 install -r requirements.txt\n   ```\n\n   - CUDA \u003e= 9.2\n   - PyTorch \u003e= 1.7.1\n   - Torchvision \u003e= 0.8.2\n   - Other required packages in `requirements.txt`\n\n## Get Started\n### Prepare pretrained models\nDownload our pretrained models from [Releases V0.1.0](https://github.com/sczhou/ProPainter/releases/tag/v0.1.0) to the `weights` folder. (All pretrained models can also be automatically downloaded during the first inference.)\n\nThe directory structure will be arranged as:\n```\nweights\n   |- ProPainter.pth\n   |- recurrent_flow_completion.pth\n   |- raft-things.pth\n   |- i3d_rgb_imagenet.pt (for evaluating VFID metric)\n   |- README.md\n```\n\n### 🏂 Quick test\nWe provide some examples in the [`inputs`](./inputs) folder. \nRun the following commands to try it out:\n```shell\n# The first example (object removal)\npython inference_propainter.py --video inputs/object_removal/bmx-trees --mask inputs/object_removal/bmx-trees_mask \n# The second example (video completion)\npython inference_propainter.py --video inputs/video_completion/running_car.mp4 --mask inputs/video_completion/mask_square.png --height 240 --width 432\n```\n\nThe results will be saved in the `results` folder.\nTo test your own videos, please prepare the input `mp4 video` (or `split frames`) and `frame-wise mask(s)`.\n\nIf you want to specify the video resolution for processing or avoid running out of memory, you can set the video size of `--width` and `--height`:\n```shell\n# process a 576x320 video; set --fp16 to use fp16 (half precision) during inference.\npython inference_propainter.py --video inputs/video_completion/running_car.mp4 --mask inputs/video_completion/mask_square.png --height 320 --width 576 --fp16\n```\n\n#### 💃🏻 Interactive Demo\n\nWe also provide an interactive demo for object removal, allowing users to select any object they wish to remove from a video. You can try the demo on [Hugging Face](https://huggingface.co/spaces/sczhou/ProPainter) or run it [locally](https://github.com/sczhou/ProPainter/tree/main/web-demos/hugging_face). \n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"./web-demos/hugging_face/assets/demo.gif\" alt=\"Demo GIF\" style=\"max-width: 512px; height: auto;\"\u003e\n\u003c/div\u003e\n\n*Please note that the demo's interface and usage may differ from the GIF animation above. For detailed instructions, refer to the [user guide](https://github.com/sczhou/ProPainter/blob/main/web-demos/hugging_face/README.md).*\n\n### 🚀 Memory-efficient inference\n\nVideo inpainting typically requires a significant amount of GPU memory. Here, we offer various features that facilitate memory-efficient inference, effectively avoiding the Out-Of-Memory (OOM) error. You can use the following options to reduce memory usage further:\n\n   - Reduce the number of local neighbors through decreasing the `--neighbor_length` (default 10).\n   - Reduce the number of global references by increasing the `--ref_stride` (default 10).\n   - Set the `--resize_ratio` (default 1.0) to resize the processing video.\n   - Set a smaller video size via specifying the `--width` and `--height`.\n   - Set `--fp16` to use fp16 (half precision) during inference.\n   - Reduce the frames of sub-videos `--subvideo_length` (default 80), which effectively decouples GPU memory costs and video length.\n\nBlow shows the estimated GPU memory requirements for different sub-video lengths with fp32/fp16 precision: \n\n| Resolution | 50 frames | 80 frames |\n| :---       | :----:    | :----:    |\n| 1280 x 720 | 28G / 19G | OOM / 25G |\n| 720 x 480  | 11G / 7G  | 13G / 8G  |\n| 640 x 480  | 10G / 6G  | 12G / 7G  |\n| 320 x 240  | 3G  / 2G  | 4G  / 3G  | \n\n\n## Dataset preparation\n\u003ctable\u003e\n\u003cthead\u003e\n  \u003ctr\u003e\n    \u003cth\u003eDataset\u003c/th\u003e\n    \u003cth\u003eYouTube-VOS\u003c/th\u003e\n    \u003cth\u003eDAVIS\u003c/th\u003e\n  \u003c/tr\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eDescription\u003c/td\u003e\n    \u003ctd\u003eFor training (3,471) and evaluation (508)\u003c/td\u003e\n    \u003ctd\u003eFor evaluation (50 in 90)\u003c/td\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eImages\u003c/td\u003e\n    \u003ctd\u003e [\u003ca href=\"https://competitions.codalab.org/competitions/19544#participate-get-data\"\u003eOfficial Link\u003c/a\u003e] (Download train and test all frames) \u003c/td\u003e\n    \u003ctd\u003e [\u003ca href=\"https://data.vision.ee.ethz.ch/csergi/share/davis/DAVIS-2017-trainval-480p.zip\"\u003eOfficial Link\u003c/a\u003e] (2017, 480p, TrainVal) \u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eMasks\u003c/td\u003e\n    \u003ctd colspan=\"2\"\u003e [\u003ca href=\"https://drive.google.com/file/d/1dFTneS_zaJAHjglxU10gYzr1-xALgHa4/view?usp=sharing\"\u003eGoogle Drive\u003c/a\u003e] [\u003ca href=\"https://pan.baidu.com/s/1JC-UKmlQfjhVtD81196cxA?pwd=87e3\"\u003eBaidu Disk\u003c/a\u003e] (For reproducing paper results; provided in \u003ca href=\"https://arxiv.org/abs/2309.03897\"\u003eProPainter\u003c/a\u003e paper) \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\n\nThe training and test split files are provided in `datasets/\u003cdataset_name\u003e`. For each dataset, you should place `JPEGImages` to `datasets/\u003cdataset_name\u003e`. Resize all video frames to size `432x240` for training. Unzip downloaded mask files to `datasets`.\n\nThe `datasets` directory structure will be arranged as: (**Note**: please check it carefully)\n```\ndatasets\n   |- davis\n      |- JPEGImages_432_240\n         |- \u003cvideo_name\u003e\n            |- 00000.jpg\n            |- 00001.jpg\n      |- test_masks\n         |- \u003cvideo_name\u003e\n            |- 00000.png\n            |- 00001.png   \n      |- train.json\n      |- test.json\n   |- youtube-vos\n      |- JPEGImages_432_240\n         |- \u003cvideo_name\u003e\n            |- 00000.jpg\n            |- 00001.jpg\n      |- test_masks\n         |- \u003cvideo_name\u003e\n            |- 00000.png\n            |- 00001.png\n      |- train.json\n      |- test.json   \n```\n\n## Training\nOur training configures are provided in [`train_flowcomp.json`](./configs/train_flowcomp.json) (for Recurrent Flow Completion Network) and [`train_propainter.json`](./configs/train_propainter.json) (for ProPainter).\n\nRun one of the following commands for training:\n```shell\n # For training Recurrent Flow Completion Network\n python train.py -c configs/train_flowcomp.json\n # For training ProPainter\n python train.py -c configs/train_propainter.json\n```\nYou can run the **same command** to **resume** your training.\n\nTo speed up the training process, you can precompute optical flow for the training dataset using the following command:\n```shell\n # Compute optical flow for training dataset\n python scripts/compute_flow.py --root_path \u003cdataset_root\u003e --save_path \u003csave_flow_root\u003e --height 240 --width 432\n```\n\n## Evaluation\nRun one of the following commands for evaluation:\n```shell\n # For evaluating flow completion model\n python scripts/evaluate_flow_completion.py --dataset \u003cdataset_name\u003e --video_root \u003cvideo_root\u003e --mask_root \u003cmask_root\u003e --save_results\n # For evaluating ProPainter model\n python scripts/evaluate_propainter.py --dataset \u003cdataset_name\u003e --video_root \u003cvideo_root\u003e --mask_root \u003cmask_root\u003e --save_results\n```\n\nThe scores and results will also be saved in the `results_eval` folder.\nPlease `--save_results` for further [evaluating temporal warping error](https://github.com/phoenix104104/fast_blind_video_consistency#evaluation).\n\n\n\n## Citation\n\n   If you find our repo useful for your research, please consider citing our paper:\n\n   ```bibtex\n   @inproceedings{zhou2023propainter,\n      title={{ProPainter}: Improving Propagation and Transformer for Video Inpainting},\n      author={Zhou, Shangchen and Li, Chongyi and Chan, Kelvin C.K and Loy, Chen Change},\n      booktitle={Proceedings of IEEE International Conference on Computer Vision (ICCV)},\n      year={2023}\n   }\n   ```\n\n\n## License\n\n#### Non-Commercial Use Only Declaration\nThe ProPainter is made available for use, reproduction, and distribution strictly for non-commercial purposes. The code and models are licensed under \u003ca rel=\"license\" href=\"./LICENSE\"\u003eNTU S-Lab License 1.0\u003c/a\u003e. Redistribution and use should follow this license.\n\nFor inquiries or to obtain permission for commercial use, please consult Dr. Shangchen Zhou (shangchenzhou@gmail.com).\n\n\n## Projects that use ProPainter\n\nIf you develop or use ProPainter in your projects, feel free to let me know. Also, please include this [ProPainter](https://github.com/sczhou/ProPainter) repo link, authorship information, and our [S-Lab license](https://github.com/sczhou/ProPainter/blob/main/LICENSE) (with link).\n\n#### Projects/Applications from the Community\n\n- Streaming ProPainter: https://github.com/osmr/propainter\n- Faster ProPainter: https://github.com/halfzm/faster-propainter\n- ProPainter WebUI: https://github.com/halfzm/ProPainter-Webui\n- ProPainter ComfyUI: https://github.com/daniabib/ComfyUI_ProPainter_Nodes\n- Cutie (video segmentation): https://github.com/hkchengrex/Cutie\n- Cinetransfer (character transfer): https://virtualfilmstudio.github.io/projects/cinetransfer\n- Motionshop (character transfer): https://aigc3d.github.io/motionshop\n\n\n\n#### PyPI\n- propainter: https://pypi.org/project/propainter\n- pytorchcv: https://pypi.org/project/pytorchcv\n\n## Contact\nIf you have any questions, please feel free to reach me out at shangchenzhou@gmail.com. \n\n## Acknowledgement\n\nThis code is based on [E\u003csup\u003e2\u003c/sup\u003eFGVI](https://github.com/MCG-NKU/E2FGVI) and [STTN](https://github.com/researchmm/STTN). Some code are brought from [BasicVSR++](https://github.com/ckkelvinchan/BasicVSR_PlusPlus). Thanks for their awesome works. \n\nSpecial thanks to [Yihang Luo](https://github.com/Luo-Yihang) for his valuable contributions to build and maintain the Gradio demos for ProPainter.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsczhou%2Fpropainter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsczhou%2Fpropainter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsczhou%2Fpropainter/lists"}