Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/sczhou/ProPainter

[ICCV 2023] ProPainter: Improving Propagation and Transformer for Video Inpainting
https://github.com/sczhou/ProPainter

object-removal video-completion video-inpainting video-outpainting watermark-removal

Last synced: 25 days ago
JSON representation

[ICCV 2023] ProPainter: Improving Propagation and Transformer for Video Inpainting

Awesome Lists containing this project

README 

        






ProPainter: Improving Propagation and Transformer for Video Inpainting





    Shangchen Zhou 

    Chongyi Li 

    Kelvin C.K. Chan 

    Chen Change Loy





    S-Lab, Nanyang Technological University  






    ICCV 2023






    


        

        

        

        

        

        

        

        

        

        

        

        

        

        

        

        

    





⭐ If ProPainter is helpful to your projects, please help star this repo. Thanks! 🤗



:open_book: For more visual results, go checkout our project page



---






## Update

- **2023.11.09**: Integrated to :man_artist: [OpenXLab](https://openxlab.org.cn/apps). Try out online demo! [![OpenXLab](https://img.shields.io/badge/Demo-%F0%9F%91%A8%E2%80%8D%F0%9F%8E%A8%20OpenXLab-blue)](https://openxlab.org.cn/apps/detail/ShangchenZhou/ProPainter)

- **2023.11.09**: Integrated to :hugs: [Hugging Face](https://huggingface.co/spaces). Try out online demo! [![Hugging Face](https://img.shields.io/badge/Demo-%F0%9F%A4%97%20Hugging%20Face-blue)](https://huggingface.co/spaces/sczhou/ProPainter)

- **2023.09.24**: We remove the watermark removal demos officially to prevent the misuse of our work for unethical purposes.

- **2023.09.21**: Add features for memory-efficient inference. Check our [GPU memory](https://github.com/sczhou/ProPainter#-memory-efficient-inference) requirements. 🚀

- **2023.09.07**: Our code and model are publicly available. 🐳

- **2023.09.01**: This repo is created.



### TODO

- [ ] Make a Colab demo.

- [x] ~~Make a interactive Gradio demo.~~

- [x] ~~Update features for memory-efficient inference.~~

  

## Results



#### 👨🏻‍🎨 Object Removal



    

      

   

    

      

   



#### 🎨 Video Completion



    

      

   

    

      

   



    

      

   

    

      

   



## Overview

![overall_structure](assets/ProPainter_pipeline.png)



## Dependencies and Installation



1. Clone Repo



   ```bash

   git clone https://github.com/sczhou/ProPainter.git

   ```



2. Create Conda Environment and Install Dependencies



   ```bash

   # create new anaconda env

   conda create -n propainter python=3.8 -y

   conda activate propainter



   # install python dependencies

   pip3 install -r requirements.txt

   ```



   - CUDA >= 9.2

   - PyTorch >= 1.7.1

   - Torchvision >= 0.8.2

   - Other required packages in `requirements.txt`



## Get Started

### Prepare pretrained models

Download our pretrained models from [Releases V0.1.0](https://github.com/sczhou/ProPainter/releases/tag/v0.1.0) to the `weights` folder. (All pretrained models can also be automatically downloaded during the first inference.)



The directory structure will be arranged as:

```

weights

   |- ProPainter.pth

   |- recurrent_flow_completion.pth

   |- raft-things.pth

   |- i3d_rgb_imagenet.pt (for evaluating VFID metric)

   |- README.md

```



### 🏂 Quick test

We provide some examples in the [`inputs`](./inputs) folder. 

Run the following commands to try it out:

```shell

# The first example (object removal)

python inference_propainter.py --video inputs/object_removal/bmx-trees --mask inputs/object_removal/bmx-trees_mask 

# The second example (video completion)

python inference_propainter.py --video inputs/video_completion/running_car.mp4 --mask inputs/video_completion/mask_square.png --height 240 --width 432

```



The results will be saved in the `results` folder.

To test your own videos, please prepare the input `mp4 video` (or `split frames`) and `frame-wise mask(s)`.



If you want to specify the video resolution for processing or avoid running out of memory, you can set the video size of `--width` and `--height`:

```shell

# process a 576x320 video; set --fp16 to use fp16 (half precision) during inference.

python inference_propainter.py --video inputs/video_completion/running_car.mp4 --mask inputs/video_completion/mask_square.png --height 320 --width 576 --fp16

```



#### 💃🏻 Interactive Demo



We also provide an interactive demo for object removal, allowing users to select any object they wish to remove from a video. You can try the demo on [Hugging Face](https://huggingface.co/spaces/sczhou/ProPainter) or run it [locally](https://github.com/sczhou/ProPainter/tree/main/web-demos/hugging_face). 





  Demo GIF




*Please note that the demo's interface and usage may differ from the GIF animation above. For detailed instructions, refer to the [user guide](https://github.com/sczhou/ProPainter/blob/main/web-demos/hugging_face/README.md).*



### 🚀 Memory-efficient inference



Video inpainting typically requires a significant amount of GPU memory. Here, we offer various features that facilitate memory-efficient inference, effectively avoiding the Out-Of-Memory (OOM) error. You can use the following options to reduce memory usage further:



   - Reduce the number of local neighbors through decreasing the `--neighbor_length` (default 10).

   - Reduce the number of global references by increasing the `--ref_stride` (default 10).

   - Set the `--resize_ratio` (default 1.0) to resize the processing video.

   - Set a smaller video size via specifying the `--width` and `--height`.

   - Set `--fp16` to use fp16 (half precision) during inference.

   - Reduce the frames of sub-videos `--subvideo_length` (default 80), which effectively decouples GPU memory costs and video length.



Blow shows the estimated GPU memory requirements for different sub-video lengths with fp32/fp16 precision: 



| Resolution | 50 frames | 80 frames |

| :---       | :----:    | :----:    |

| 1280 x 720 | 28G / 19G | OOM / 25G |

| 720 x 480  | 11G / 7G  | 13G / 8G  |

| 640 x 480  | 10G / 6G  | 12G / 7G  |

| 320 x 240  | 3G  / 2G  | 4G  / 3G  | 



## Dataset preparation



  

    Dataset

    YouTube-VOS

    DAVIS

  



  

    Description

    For training (3,471) and evaluation (508)

    For evaluation (50 in 90)

  

    Images

     [Official Link] (Download train and test all frames) 

     [Official Link] (2017, 480p, TrainVal) 

  

  

    Masks

     [Google Drive] [Baidu Disk] (For reproducing paper results; provided in ProPainter paper) 

  



The training and test split files are provided in `datasets/`. For each dataset, you should place `JPEGImages` to `datasets/`. Resize all video frames to size `432x240` for training. Unzip downloaded mask files to `datasets`.



The `datasets` directory structure will be arranged as: (**Note**: please check it carefully)

```

datasets

   |- davis

      |- JPEGImages_432_240

         |- 

            |- 00000.jpg

            |- 00001.jpg

      |- test_masks

         |- 

            |- 00000.png

            |- 00001.png   

      |- train.json

      |- test.json

   |- youtube-vos

      |- JPEGImages_432_240

         |- 

            |- 00000.jpg

            |- 00001.jpg

      |- test_masks

         |- 

            |- 00000.png

            |- 00001.png

      |- train.json

      |- test.json   

```



## Training

Our training configures are provided in [`train_flowcomp.json`](./configs/train_flowcomp.json) (for Recurrent Flow Completion Network) and [`train_propainter.json`](./configs/train_propainter.json) (for ProPainter).



Run one of the following commands for training:

```shell

 # For training Recurrent Flow Completion Network

 python train.py -c configs/train_flowcomp.json

 # For training ProPainter

 python train.py -c configs/train_propainter.json

```

You can run the **same command** to **resume** your training.



To speed up the training process, you can precompute optical flow for the training dataset using the following command:

```shell

 # Compute optical flow for training dataset

 python scripts/compute_flow.py --root_path  --save_path  --height 240 --width 432

```



## Evaluation

Run one of the following commands for evaluation:

```shell

 # For evaluating flow completion model

 python scripts/evaluate_flow_completion.py --dataset  --video_root  --mask_root  --save_results

 # For evaluating ProPainter model

 python scripts/evaluate_propainter.py --dataset  --video_root  --mask_root  --save_results

```



The scores and results will also be saved in the `results_eval` folder.

Please `--save_results` for further [evaluating temporal warping error](https://github.com/phoenix104104/fast_blind_video_consistency#evaluation).



## Citation



   If you find our repo useful for your research, please consider citing our paper:



   ```bibtex

   @inproceedings{zhou2023propainter,

      title={{ProPainter}: Improving Propagation and Transformer for Video Inpainting},

      author={Zhou, Shangchen and Li, Chongyi and Chan, Kelvin C.K and Loy, Chen Change},

      booktitle={Proceedings of IEEE International Conference on Computer Vision (ICCV)},

      year={2023}

   }

   ```



## License



#### Non-Commercial Use Only Declaration

The ProPainter is made available for use, reproduction, and distribution strictly for non-commercial purposes. The code and models are licensed under NTU S-Lab License 1.0. Redistribution and use should follow this license.



For inquiries or to obtain permission for commercial use, please consult Dr. Shangchen Zhou ([email protected]).



## Projects that use ProPainter



If you develop or use ProPainter in your projects, feel free to let me know. Also, please include this [ProPainter](https://github.com/sczhou/ProPainter) repo link, authorship information, and our [S-Lab license](https://github.com/sczhou/ProPainter/blob/main/LICENSE) (with link).



#### Projects/Applications from the Community



- Streaming ProPainter: https://github.com/osmr/propainter

- Faster ProPainter: https://github.com/halfzm/faster-propainter

- ProPainter WebUI: https://github.com/halfzm/ProPainter-Webui

- ProPainter ComfyUI: https://github.com/daniabib/ComfyUI_ProPainter_Nodes

- Cutie (video segmentation): https://github.com/hkchengrex/Cutie

- Cinetransfer (character transfer): https://virtualfilmstudio.github.io/projects/cinetransfer

- Motionshop (character transfer): https://aigc3d.github.io/motionshop



#### PyPI

- propainter: https://pypi.org/project/propainter

- pytorchcv: https://pypi.org/project/pytorchcv



## Contact

If you have any questions, please feel free to reach me out at [email protected]. 



## Acknowledgement



This code is based on [E2FGVI](https://github.com/MCG-NKU/E2FGVI) and [STTN](https://github.com/researchmm/STTN). Some code are brought from [BasicVSR++](https://github.com/ckkelvinchan/BasicVSR_PlusPlus). Thanks for their awesome works. 



Special thanks to [Yihang Luo](https://github.com/Luo-Yihang) for his valuable contributions to build and maintain the Gradio demos for ProPainter.