https://codegoat24.github.io/MagicFace/

Official implementation of MagicFace: Training-free Universal-Style Human Image Customized Synthesis.
https://codegoat24.github.io/MagicFace/

Last synced: about 1 month ago
JSON representation

Official implementation of MagicFace: Training-free Universal-Style Human Image Customized Synthesis.

• Host: GitHub
• URL: https://codegoat24.github.io/MagicFace/
• Owner: CodeGoat24
• Created: 2024-08-14T05:06:26.000Z (about 1 month ago)
• Default Branch: main
• Last Pushed: 2024-08-16T05:27:41.000Z (about 1 month ago)
• Last Synced: 2024-08-16T06:36:34.856Z (about 1 month ago)
• Homepage: https://codegoat24.github.io/MagicFace/
• Size: 19.3 MB
• Stars: 6
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

• awesome-diffusion-categorized - [Project

README

        


MagicFace: Training-free Universal-Style Human Image Customized Synthesis


[Yibin Wang](https://codegoat24.github.io)\*, [Weizhong Zhang](https://weizhonz.github.io/)\*, [Cheng Jin](https://cjinfdu.github.io/)† 

(*equal contribution, †corresponding author)

[Fudan University]











![teaser](docs/static/images/teaser.png)

## 📖 Abstract



Existing human image personalized generation methods often require tedious training: either fine-tuning with a few images or retraining on large-scale datasets. 

In such cases, these methods are prone to overfitting and encounter difficulties when personalizing individuals of diverse styles. Moreover, these training-based approaches also struggle with multi-concept human image customizing. To this end, we propose MagicFace, the first method for universal-style human image personalized synthesis that enables single/multi-concept customization for humans of any style in a training-free manner. MagicFace introduces a coarse-to-fine generation pipeline, involving two sequential stages: semantic scene construction and concept feature injection. This is achieved by our Reference-aware Self-Attention (RSA) and Region-grouped Blend Attention (RBA) mechanisms. Specifically, 

in the first stage, RSA enables the latent image to query features from reference concepts simultaneously, extracting the coarse-grained overall semantic understanding to facilitate the initial semantic layout establishment. 

In the second stage, we employ a training-free semantic segmentation method to pinpoint the generated regions of all concepts in the latent image at each step. Following this, RBA divides the pixels of the latent image into semantic groups, with each group querying fine-grained features from its reference concept, which ensures precise attribute alignment and feature injection. Throughout the two-stage process, a weight mask strategy is employed to ensure the model focuses more on the reference concepts. 

Extensive experiments demonstrate our superiority in both human-centric subject-to-image synthesis and multi-concept human image customization. Our approach also can be applied to texture transformation, further enhancing its versatility and applicability.



![architecture](docs/static/images/architecture.png)

## 🗓️ TODO

- [ ] Release inference code

- [ ] Release demo

- [ ] Release evaluation datasets

- [ ] Release evaluation code

- [ ] Release visualization code

## 🖼️ Visual results of MagicFace

![visual_result](figures/visual_result_photorealism.png)

![visual_result](figures/visual_result_diverse_style.png)