https://codegoat24.github.io/MagicFace/

Official implementation of MagicFace: Training-free Universal-Style Human Image Customized Synthesis.
https://codegoat24.github.io/MagicFace/

Last synced: about 1 month ago
JSON representation

Official implementation of MagicFace: Training-free Universal-Style Human Image Customized Synthesis.

• Host: GitHub
• URL: https://codegoat24.github.io/MagicFace/
• Owner: CodeGoat24
• Created: 2024-08-14T05:06:26.000Z (5 months ago)
• Default Branch: main
• Last Pushed: 2024-11-18T03:50:26.000Z (about 2 months ago)
• Last Synced: 2024-11-18T04:37:47.757Z (about 2 months ago)
• Homepage: https://codegoat24.github.io/MagicFace/
• Size: 33.6 MB
• Stars: 51
• Watchers: 5
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

• awesome-diffusion-categorized - [Project

README

        


MagicFace: Training-free Universal-Style Human Image Customized Synthesis


[Yibin Wang](https://codegoat24.github.io), [Weizhong Zhang](https://weizhonz.github.io/)†, [Cheng Jin](https://cjinfdu.github.io/)† 

(†corresponding author)

[Fudan University]











![teaser](docs/static/images/teaser.png)

## Release

- [2024/11/18] 🔥 We update the figures and include inference time comparisons in the [paper](https://arxiv.org/pdf/2408.07433).

- [2024/08/15] 🔥 We release the [paper](https://arxiv.org/pdf/2408.07433).

- [2024/08/14] 🔥 We launch the [project page](https://codegoat24.github.io/MagicFace/).

## 📖 Abstract



Current human image customization methods leverage Stable Diffusion (SD) for its rich semantic prior. 

However, since SD is not specifically designed for human-oriented generation, these methods often require extensive fine-tuning on large-scale datasets, which renders them susceptible to overfitting and hinders their ability to personalize individuals with previously unseen styles.

Moreover, these methods extensively focus on single-concept human image synthesis and lack the flexibility to customize individuals using multiple given concepts, thereby impeding their broader practical application.

This paper proposes MagicFace, a novel training-free method for multi-concept universal-style human image personalized synthesis. 

Our core idea is to simulate how humans create images given specific concepts, i.e., first establish a semantic layout considering factors such as concepts' shape and posture, then optimize details by comparing with concepts at the pixel level. To implement this process, we introduce a coarse-to-fine generation pipeline, involving two sequential stages: semantic layout construction and concept feature injection. This is achieved by our Reference-aware Self-Attention (RSA) and Region-grouped Blend Attention (RBA) mechanisms.  

In the first stage, RSA enables the latent image to query features from all reference concepts simultaneously, extracting the overall semantic understanding to facilitate the initial semantic layout establishment. 

In the second stage, we employ an attention-based semantic segmentation method to pinpoint the latent generated regions of all concepts at each step. Following this, RBA divides the pixels of the latent image into semantic groups, with each group querying fine-grained features from the corresponding reference concept.

Notably, our method empowers users to freely control the influence of each concept on customization through a weighted mask strategy.

Extensive experiments demonstrate the superiority of MagicFace in both single- and multi-concept human image customization. 



![architecture](docs/static/images/architecture.png)

## 🗓️ TODO

- [ ] Release inference code

- [ ] Release demo

- [ ] Release evaluation datasets

- [ ] Release evaluation code

- [ ] Release visualization code

## 🖼️ Visual results of MagicFace

![visual_result](docs/static/images/visual_result_photorealism.png)

![visual_result](docs/static/images/visual_result_diverse_style.png)