https://github.com/wangzhiyaoo/SVFR

Official implementation of SVFR.
https://github.com/wangzhiyaoo/SVFR

Last synced: 6 days ago
JSON representation

Official implementation of SVFR.

• Host: GitHub
• URL: https://github.com/wangzhiyaoo/SVFR
• Owner: wangzhiyaoo
• Created: 2024-12-17T12:01:14.000Z (25 days ago)
• Default Branch: main
• Last Pushed: 2025-01-03T03:19:30.000Z (9 days ago)
• Last Synced: 2025-01-03T03:30:42.025Z (9 days ago)
• Language: Python
• Homepage: https://wangzhiyaoo.github.io/SVFR/
• Size: 1.95 KB
• Stars: 3
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

• awesome-diffusion-categorized - [Code

README

        




SVFR: A Unified Framework for Generalized Video Face Restoration



[![arXiv](https://img.shields.io/badge/arXiv-2307.04725-b31b1b.svg)](https://arxiv.org/pdf/2501.01235)

[![Project Page](https://img.shields.io/badge/Project-Website-green)](https://wangzhiyaoo.github.io/SVFR/)

## 🔥 Overview

SVFR is a unified framework for face video restoration that supports tasks such as **BFR, Colorization, Inpainting**, and **their combinations** within one cohesive system.



## 🎬 Demo

### BFR

| Case1                                                                                                                        | Case2                                                                                                                        |

|--------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------|

| |  |

### BFR+Colorization

| Case3                                                                                                                        | Case4                                                                                                                        |

|--------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------|

| |  |

### BFR+Colorization+Inpainting

| Case5                                                                                                                        | Case6                                                                                                                        |

|--------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------|

| |  |

## 🎙️ News

- **[2025.01.02]**: We released the initial version of the [inference code](#inference) and [models](#download-checkpoints). Stay tuned for continuous updates!

- **[2024.12.17]**: This repo is created!

## 🚀 Getting Started

## Setup

Use the following command to install a conda environment for SVFR from scratch:

```bash

conda create -n svfr python=3.9 -y

conda activate svfr

```

Install PyTorch:  make sure to select the appropriate CUDA version based on your hardware, for example,

```bash

pip install torch==2.2.2 torchvision==0.17.2 torchaudio==2.2.2

```

Install Dependencies:

```bash

pip install -r requirements.txt

```

## Download checkpoints

Download the Stable Video Diffusion


```

conda install git-lfs

git lfs install

git clone https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt models/stable-video-diffusion-img2vid-xt

```

Download SVFR


You can download checkpoints manually through link on [Google Drive](https://drive.google.com/drive/folders/1nzy9Vk-yA_DwXm1Pm4dyE2o0r7V6_5mn?usp=share_link).

Put checkpoints as follows:

```

└── models

    ├── face_align

    │   ├── yoloface_v5m.pt

    ├── face_restoration

    │   ├── unet.pth

    │   ├── id_linear.pth

    │   ├── insightface_glint360k.pth

    └── stable-video-diffusion-img2vid-xt

        ├── vae

        ├── scheduler

        └── ...

```

## Inference

### Inference single or multi task

```

python3 infer.py \

 --config config/infer.yaml \

 --task_ids 0 \

 --input_path ./assert/lq/lq1.mp4 \

 --output_dir ./results/ 

```

task_id:


> 0 -- bfr  

> 1 -- colorization  

> 2 -- inpainting  

> 0,1 -- bfr and colorization  

> 0,1,2 -- bfr and colorization and inpainting  

> ...

### Inference with additional inpainting mask

```

# For Inference with Inpainting

# Add '--mask_path' if you need to specify the mask file.

python3 infer.py \

 --config config/infer.yaml \

 --task_ids 0,1,2 \

 --input_path ./assert/lq/lq3.mp4 \

 --output_dir ./results/ 

 --mask_path ./assert/mask/lq3.png

```

## BibTex

```

@misc{wang2025svfrunifiedframeworkgeneralized,

      title={SVFR: A Unified Framework for Generalized Video Face Restoration}, 

      author={Zhiyao Wang and Xu Chen and Chengming Xu and Junwei Zhu and Xiaobin Hu and Jiangning Zhang and Chengjie Wang and Yuqi Liu and Yiyi Zhou and Rongrong Ji},

      year={2025},

      eprint={2501.01235},

      archivePrefix={arXiv},

      primaryClass={cs.CV},

      url={https://arxiv.org/abs/2501.01235}, 

}

```