Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/DevilPG/AAAI2024-CPABMM
[AAAI 2024] Continuous Piecewise-Affine Based Motion Model for Image Animation
https://github.com/DevilPG/AAAI2024-CPABMM
image-animation motion-transfer video-generation
Last synced: 2 months ago
JSON representation
[AAAI 2024] Continuous Piecewise-Affine Based Motion Model for Image Animation
- Host: GitHub
- URL: https://github.com/DevilPG/AAAI2024-CPABMM
- Owner: DevilPG
- Created: 2023-12-12T11:23:46.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2024-01-19T16:08:24.000Z (about 1 year ago)
- Last Synced: 2024-01-20T11:55:09.562Z (about 1 year ago)
- Topics: image-animation, motion-transfer, video-generation
- Language: Python
- Homepage:
- Size: 3.71 MB
- Stars: 5
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- Awesome-Segment-Anything - [code
README
# Continuous Piecewise-Affine Based Motion Model for Image Animation (AAAI 2024)
### [Paper](https://doi.org/10.48550/arXiv.2401.09146 )
[Hexiang Wang](https://github.com/DevilPG),
[Fengqi Liu]([email protected]),
[Qianyu Zhou](https://qianyuzqy.github.io/),
[Ran Yi](https://yiranran.github.io/),
[Xin Tan](https://tanxincs.github.io/),
and [Lizhuang Ma](https://dmcv.sjtu.edu.cn/)
## Introduction
### Abstract
>Image animation aims to bring static images to life according to driving videos and create engaging visual content that can be used for various purposes such as animation, entertainment, and education. Recent unsupervised methods utilize affine and thin-plate spline transformations based on keypoints to transfer the motion in driving frames to the source image. However, limited by the expressive power of the transformations used, these methods always produce poor results when the gap between the motion in the driving frame and the source image is large. To address this issue, we propose to model motion from the source image to the driving frame in highly-expressive diffeomorphism spaces. Firstly, we introduce Continuous Piecewise-Affine based (CPAB) transformation to model the motion and present a well-designed inference algorithm to generate CPAB transformation from control keypoints. Secondly, we propose a SAM-guided keypoint semantic loss to further constrain the keypoint extraction process and improve the semantic consistency between the corresponding keypoints on the source and driving images. Finally, we design a structure alignment loss to align the structure-related features extracted from driving and generated images, thus helping the generator generate results
that are more consistent with the driving action. Extensive experiments on four datasets demonstrate the effectiveness of our method against state-of-the-art competitors quantitatively and qualitatively.
This work has been accepted by AAAI 2024.
### Examples of image animation
## Installation
Clone this project in recursive mode:
```bash
git clone --recursive https://github.com/DevilPG/AAAI2024-CPABMM.git
```
For detailed information about the CPAB transformation library, please refer to [libcpab](https://github.com/SkafteNicki/libcpab.git).
Please run the project in ```python3``` environment.
To install the dependencies run:
```bash
pip install -r requirements.txt
```
## Configuration
Configuration files are in the `config` folder, one for each `dataset`, named as ```config/dataset_name.yaml```.
## Datasets
For detailed information of four datasets used in this paper, please follow [TPSMM](https://github.com/yoyo-nb/Thin-Plate-Spline-Motion-Model).
## Training
To train a model on specific dataset run:
```
python run.py --config config/dataset_name.yaml --log_dir dir_to_save_log_files --tag id_for_training_task
```
### Training AVD network
To train a model on specific dataset run:
```
python run.py --mode train_avd --checkpoint '{checkpoint_folder}/checkpoint.pth.tar' --config config/dataset_name.yaml
```
## Evaluation on video reconstruction
To evaluate the model on video reconstruction task run:
```
CUDA_VISIBLE_DEVICES=0 python run.py --mode reconstruction --config config/dataset_name.yaml --checkpoint '{checkpoint_folder}/checkpoint.pth.tar'
```
To compute quantitative metrics, please follow instructions from [pose-evaluation](https://github.com/AliaksandrSiarohin/pose-evaluation).
## Image animation demo
To generate a animation video with a source image and a driving video run:
```bash
python demo.py --config config/dataset_name.yaml --checkpoint {checkpoint_folder}/checkpoint.pth.tar --source_image path_to_source_img --driving_video path_to_driving_video
```
## Citation
If you find this code helpful for your research, please cite:
```
@article{wang2024cpabmm,
title={Continuous Piecewise-Affine Based Motion Model for Image Animation},
author={Hexiang Wang and Fengqi Liu and Qianyu Zhou and Ran Yi and Xin Tan and Lizhuang Ma},
journal={arXiv preprint arXiv:2401.09146},
year={2024}
}
```