Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/ali-vilab/ACE_plus


https://github.com/ali-vilab/ACE_plus

Last synced: about 1 month ago
JSON representation

Awesome Lists containing this project

README 

        



  


 ++: Instruction-Based Image Creation and Editing 
 via Context-Aware Content Filling 



  


    Paper PDF

    Project Page

    

    

    

    

    


    Chaojie Mao

    ·

    Jingfeng Zhang

    ·

    Yulin Pan

    ·

    Zeyinzi Jiang

    ·

    Zhen Han

    


    ·

    Yu Liu

    ·

    Jingren Zhou

    


    Tongyi Lab, Alibaba Group

  


  

    

    

      

    

    

  



## 📚 Introduction



The original intention behind the design of ACE++ was to unify reference image generation, local editing, 

and controllable generation into a single framework, and to enable one model to adapt to a wider range of tasks. 

A more versatile model is often capable of handling more complex tasks. We have released three LoRA models for 

specific vertical domains and a more versatile FFT model (the performance of the FFT model may decline compared 

to the LoRA model across various tasks). Users can flexibly utilize these models and their 

combinations for their own scenarios. Furthermore, many community members have found that using them 

in conjunction with Redux modules significantly improves performance. We believe there are many more 

use cases to explore.



## 📢 News

- [x] **[2025.01.06]** Release the code and models of ACE++.

- [x] **[2025.01.07]** Release the demo on [HuggingFace](https://huggingface.co/spaces/scepter-studio/ACE-Plus).

- [x] **[2025.01.16]** Release the training code for lora.

- [x] **[2025.02.15]** Collection of workflows in Comfyui.

- [x] **[2025.02.15]** Release the config for fully fine-tuning.

- [x] **[2025.03.03]** Release a unified fft model for ACE++, support more image to image tasks.

- [x] **[2025.03.11]** Release the comfyui workflow for ACE++ FFT model. 



- We sincerely apologize 

for the delayed responses and updates regarding ACE++ issues. 

Further development of the ACE model through post-training on the FLUX model must be suspended. 

We have identified several significant challenges in post-training on the FLUX foundation. 

The primary issue is the high degree of heterogeneity between the training dataset and the FLUX model, 

which results in highly unstable training. Moreover, FLUX-Dev is a distilled model, and the influence of its original negative prompts on its final performance is uncertain.

As a result, subsequent efforts will be focused on post-training the ACE model using the Wan series of foundational models.



- __I would like to emphasize that, due to the reasons mentioned earlier, the performance of the FFT model may decline compared 

to the LoRA model across various tasks. Therefore, we recommend continuing to use the LoRA model to achieve better results. 

We provide the FFT model with the hope that it may facilitate academic exploration and research in this area.__



- We have been busy with other projects recently. Our new work in the video domain, [VACE](https://ali-vilab.github.io/VACE-Page/), 

has also been released, and we welcome you to continue following our work.



## 🔥The unified fft model for ACE++

Fully finetuning a composite model with ACE’s data to support various editing and reference generation tasks through an instructive approach.



We found that there are conflicts between the repainting task and the editing task during the experimental process. This is because the edited image is concatenated with noise in the channel dimension, whereas the repainting task modifies the region using zero pixel values in the VAE's latent space. The editing task uses RGB pixel values in the modified region through the VAE's latent space, which is similar to the distribution of the non-modified part of the repainting task, making it a challenge for the model to distinguish between the two tasks.



To address this issue, we introduced 64 additional channels in the channel dimension to differentiate between these two tasks. In these channels, we place the latent representation of the pixel space from the edited image, while keeping other channels consistent with the repainting task. This approach significantly enhances the model's adaptability to different tasks.



One issue with this approach is that it changes the input channel number of the FLUX-Fill-Dev model from 384 to 448. The specific configuration can be referenced in the [configuration file](config/ace_plus_fft.yaml).



We used tools from [stella](https://gist.github.com/Stella2211/10f5bd870387ec1ddb9932235321068e) to convert the fft-fp16 model to fft-fp8. 

But We have tested with some users and found that the FP8 version has precision loss, so we will not provide the FP8 version of the model for the time being.



### ComfyUI Workflow



Copy the workflow/ComfyUI-ACE_Plus folder into ComfyUI’s custom_nodes directory. Launch ComfyUI, and we have provided four example workflows in workflow_example_fft with the following explanations.



We provide a parameter to adjust the GPU memory usage. As shown in the figure below, max_seq_length controls the length of the token sequence during inference, thereby controlling the model's inference memory consumption. 

The range of this value is from 1024 to 5120, and it correspondingly affects the clarity of the generated image. The smaller the value, the lower the image clarity.






  

    Workflow

    Description

    Setting

  

  

    ACE_Plus_FFT_workflow_no_preprocess.json

    Use the preprocessed images, such as depth and contour, as input, or the super-resolution.

    Task_type: no_preprocess (you don't need to install dependencies like scepter)

  

  

    ACE_Plus_FFT_workflow_controlpreprocess.json

    Controllable image-to-image translation capability. To preprocess depth and contour information from images, 

    we use externally-provided models that are typically downloaded from the ModelScope Hub. Because download success 

    can vary depending on the user's environment, we offer alternatives: users can either leverage existing community 

    nodes (depth extration node or contour extraction node) for this task (then choosing the 'no_preprocess' option), 

    or users can pre-download the required models 

    contour and

    depth    

    and adjust

    the configuration file 'workflow/ComfyUI-ACE_Plus/config/ace_plus_fft_processor.yaml' to

    specify the models' local paths.

    Task_type: contour_repainting/depth_repainting/recolorizing (you need to install dependencies like scepter)

  

  

    ACE_Plus_FFT_workflow_reference_generation.json

    Reference image generation capability for portrait or subject.

    Task_type: repainting (you don't need to install dependencies like scepter)

  

  

    ACE_Plus_FFT_workflow_referenceediting_generation.json

    Reference image editing capability

    Task_type: repainting (you don't need to install dependencies like scepter)

  

 



### Examples



  

    Input Reference Image

    Input Edit Image

    Input Edit Mask

    Output

    Instruction

    Function

  

  

    

    

    

    

    "Maintain the facial features, A girl is wearing a neat police uniform and sporting a badge. She is smiling with a friendly and confident demeanor. The background is blurred, featuring a cartoon logo."

    "Character ID Consistency Generation"

  

  

    

    

        

    

    "Display the logo in a minimalist style printed in white on a matte black ceramic coffee mug, alongside a steaming cup of coffee on a cozy cafe table."

    "Subject Consistency Generation"

  

  

    

    

    

    

    "The item is put on the table."

    "Subject Consistency Editing"

  

  

    

    

    

    

    "The logo is printed on the headphones."

    "Subject Consistency Editing"

  

  

    

    

    

    

    "The woman dresses this skirt."

    "Try On"

  

  

    

    

    

    

    "{image}, the man faces the camera."

    "Face swap"

  

 

    

    

    

    

    "{image} features a close-up of a young, furry tiger cub on a rock. The tiger, which appears to be quite young, has distinctive orange, black, and white striped fur, typical of tigers. The cub's eyes have a bright and curious expression, and its ears are perked up, indicating alertness. The cub seems to be in the act of climbing or resting on the rock. The background is a blurred grassland with trees, but the focus is on the cub, which is vividly colored while the rest of the image is in grayscale, drawing attention to the tiger's details. The photo captures a moment in the wild, depicting the charming and tenacious nature of this young tiger, as well as its typical interaction with the environment."

    "Super-resolution"

  

  

    

    

    

    

    "a blue hand"

    "Regional Editing"

  

  

    

    

    

    

    "Mechanical  hands like a robot"

    "Regional Editing"

  

  

    

    

    

    

    "{image} Beautiful female portrait, Robot with smooth White transparent carbon shell, rococo detailing, Natural lighting, Highly detailed, Cinematic, 4K."

    "Recolorizing"

  

  

    

    

    

    

    "{image} Beautiful female portrait, Robot with smooth White transparent carbon shell, rococo detailing, Natural lighting, Highly detailed, Cinematic, 4K."

    "Depth Guided Generation"

  

  

    

    

    

    

    "{image} Beautiful female portrait, Robot with smooth White transparent carbon shell, rococo detailing, Natural lighting, Highly detailed, Cinematic, 4K."

    "Contour Guided Generation"

  



##  Comfyui Workflows in community

We are deeply grateful to the community developers for building many fascinating applications based on the ACE++ series of models. 

During this process, we have received valuable feedback, particularly regarding artifacts in generated images and the stability of the results. 

In response to these issues, many developers have proposed creative solutions, which have greatly inspired us, and we pay tribute to them. 

At the same time, we will take these concerns into account in our further optimization efforts, carefully evaluating and testing before releasing new models.



In the table below, we have briefly listed some workflows for everyone to use.



  

    Workflow

    Author

    Example

  

  

     【飞翔鲸】王炸！FLUX+ACE++换头  

     飞翔鲸 

    

  

  

     最新ACE++ Redux 万物迁移 AI模特换装 UP 楚门的AI世界  

     楚门的AI世界 

    

  

  

     Ace Plus+Fill+Redux稳定无抽卡换装工作流 

     T8star-Aix 

    

  

  

     Ace Plus+Fill+Redux稳定少抽卡标志工作流 

     T8star-Aix 

    

  

  

     Ace Plus+Fill+Redux稳定无抽卡换脸工作流 

     T8star-Aix 

    

  

  

     ace++ face swap in different styles  

     jax 

    

  

  

     fllux ace++ subject without reference image  

     leeguandong 

    

  

  

     Scepter-ACE++ More convenient replacement of everything 

     HaoBeen 

    

  



Additionally, many bloggers have published tutorials on how to use it, which are listed in the table below.



  

    Tutorial

  

  

     Best Faceswapper I've Seen. ACE++ in ComfyUI.  

  

  

     ACE ++ In ComfyUI All-round Creator & Editor - More Than Just A Faceswap AI  

  

  

     Ai绘画进阶140-咦？大家用的都不对？！Ace Plus工作流正确搭建方式及逻辑，参数详解，Flux Fill，Redux联用-T8 Comfyui教程 

  

  

     ace++：告别 Lora 训练，无需pulid，轻松打造专属角色！ | No Lora Training, Easily Create Exclusive Characters! 

  

  

     Ace++ and Flux Fill: Advanced Face Swapping Made Easy in ComfyUI | No Lora Training, Easily Create Exclusive Characters! 

  

  

     ComfyUI - ACE Plus Subject Portrait Lora  

  

  

     🤗AI一致性技术新突破！ACE++技术一致性comfyui工作流🍋‍  

  

  

     ComfyUI 第55集 人像换脸 FLUX的FILL模型+ACE LORA  

  

  

     换装变脸贴logo，无所不能的Ace_Plus lora  

  



### ACE++ Portrait

Portrait-consistent generation to maintain the consistency of the portrait.



  

    Tuning Method

    Input

    Output

    Instruction

    Models

  

  

    LoRA 
+ ACE Data

    

    

    "Maintain the facial features. A girl is wearing a neat police uniform and sporting a badge. She is smiling with a friendly and confident demeanor. The background is blurred, featuring a cartoon logo."

    ModelScope link  HuggingFace link  

  



Models' scepter_path: 

- **ModelScope:** ms://iic/ACE_Plus@portrait/xxxx.safetensors

- **HuggingFace:** hf://ali-vilab/ACE_Plus@portrait/xxxx.safetensors



### ACE++ Subject

Subject-driven image generation task to maintain the consistency of a specific subject in different scenes.



  

    Tuning Method

    Input

    Output

    Instruction

    Models

  

  

    LoRA 
+ ACE Data

    

    

    "Display the logo in a minimalist style printed in white on a matte black ceramic coffee mug, alongside a steaming cup of coffee on a cozy cafe table."

    ModelScope link  HuggingFace link  

  



Models' scepter_path: 

- **ModelScope:** ms://iic/ACE_Plus@subject/xxxx.safetensors

- **HuggingFace:** hf://ali-vilab/ACE_Plus@subject/xxxx.safetensors



### ACE++ LocalEditing

Redrawing the mask area of images while maintaining the original structural information of the edited area.



  

    Tuning Method

    Input

    Output

    Instruction

    Models

  

  

    LoRA 
+ ACE Data

    


    

    "By referencing the mask, restore a partial image from the doodle {image} that aligns with the textual explanation: "1 white old owl"."

    ModelScope link  HuggingFace link  

  



Models' scepter_path: 

- **ModelScope:** ms://iic/ACE_Plus@local_editing/xxxx.safetensors

- **HuggingFace:** hf://ali-vilab/ACE_Plus@local_editing/xxxx.safetensors



##  🔥 Applications

The ACE++ model supports a wide range of downstream tasks through simple adaptations. Here are some examples, and we look forward to seeing the community explore even more exciting applications utilizing the ACE++ model.



  

    Application

    ACE++ Model

    Examples

  

  

    Try On

    ACE++ Subject

    

    

    

    

    "The woman dresses this skirt."

  

  

    Logo Paste

    ACE++ Subject

    

    

    

    

    "The logo is printed on the headphones."

  

  

    Photo Editing

    ACE++ Subject

    

    

    

    

    "The item is put on the ground."

  

  

    Movie Poster Editor

    ACE++ Portrait

    

    

    

    

    "The man is facing the camera and is smiling."

  



## ⚙️️ Installation

Download the code using the following command:

```bash

git clone https://github.com/ali-vilab/ACE_plus.git

```



Install the necessary packages with `pip`: 

```bash

cd ACE_plus

pip install -r repo_requirements.txt

```

ACE++ depends on FLUX.1-Fill-dev as its base model, which you can download from [![HuggingFace link](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model-yellow)](https://huggingface.co/black-forest-labs/FLUX.1-Fill-dev). 

In order to run the inference code or Gradio demo normally, we have defined the relevant environment variables to specify the location of the model. 

For model preparation, we provide three methods for downloading the model. The summary of relevant settings is as follows.



|   Model Downloading Method    | Clone to Local Path                                                                                                                                                                                                                                         | Automatic Downloading during Runtime
(Setting the Environment Variables using scepter_path in [ACE Models](#-ace-models))                                                                                                       |

|:-----------------------------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|

| Environment Variables Setting | 
export FLUX_FILL_PATH="path/to/FLUX.1-Fill-dev"
export PORTRAIT_MODEL_PATH="path/to/ACE++ PORTRAIT PATH"
export SUBJECT_MODEL_PATH="path/to/ACE++ SUBJECT PATH"
export LOCAL_MODEL_PATH="path/to/ACE++ LOCAL EDITING PATH"
 | 
export FLUX_FILL_PATH="hf://black-forest-labs/FLUX.1-Fill-dev"
export PORTRAIT_MODEL_PATH="${scepter_path}"
export SUBJECT_MODEL_PATH="${scepter_path}"
export LOCAL_MODEL_PATH="${scepter_path}"
 |



## 🚀 Inference

Under the condition that the environment variables defined in [Installation](#-installation), users can run examples and test your own samples by executing infer.py. 

The relevant commands for lora models are as follows:

```bash

export FLUX_FILL_PATH="hf://black-forest-labs/FLUX.1-Fill-dev"

export PORTRAIT_MODEL_PATH="ms://iic/ACE_Plus@portrait/comfyui_portrait_lora64.safetensors"                                                                                                                                      

export SUBJECT_MODEL_PATH="ms://iic/ACE_Plus@subject/comfyui_subject_lora16.safetensors"                                                                                                                                         

export LOCAL_MODEL_PATH="ms://iic/ACE_Plus@local_editing/comfyui_local_lora16.safetensors" 

# Use the model from huggingface

# export PORTRAIT_MODEL_PATH="hf://ali-vilab/ACE_Plus@portrait/comfyui_portrait_lora64.safetensors"        

# export SUBJECT_MODEL_PATH="hf://ali-vilab/ACE_Plus@subject/comfyui_subject_lora16.safetensors"        

# export LOCAL_MODEL_PATH="hf://ali-vilab/ACE_Plus@local_editing/comfyui_local_lora16.safetensors" 

python infer_lora.py

```

The relevant commands for fft models are as follows:

```bash

export FLUX_FILL_PATH="hf://black-forest-labs/FLUX.1-Fill-dev"

export ACE_PLUS_FFT_MODEL="ms://iic/ACE_Plus@ace_plus_fft.safetensors.safetensors"                                                                                                                                      

python infer_fft.py

```



## 🚀 Train

We provide training code that allows users to train on their own data. Reference the data in 'data/train.csv' and 'data/eval.csv' to construct the training data and test data, respectively. We use '#;#' to separate fields. 

The required fields include the following six, with their explanations as follows.

```angular2html

"edit_image": represents the input image for the editing task. If it is not an editing task but a reference generation, this field can be left empty.

"edit_mask": represents the input image mask for the editing task, used to specify the editing area. If it is not an editing task but rather for reference generation, this field can be left empty.

"ref_image": represents the input image for the reference image generation task; if it is a pure editing task, this field can be left empty.

"target_image": represents the generated target image and cannot be empty.

"prompt": represents the prompt for the generation task.

"data_type": represents the type of data, which can be 'portrait', 'subject', or 'local'. This field is not used in training phase.

```



All parameters related to training are stored in 'train_config/ace_plus_lora.yaml'. With the following default configuration, the memory usage for LoRA training is between 38GB and 40GB. 



| Hyperparameter | Value     | Description |

| --- | --- | --- |

| ATTN_BACKEND |   flash_attn / pytorch  |Set 'flash_attn' to use flash_attn2(Make sure you have installed flash-attn2 correctly). If the version of PyTorch is greater than 2.4.0, use 'pytorch' to utilize PyTorch's implementation.|

| USE_GRAD_CHECKPOINT |   True / False  |Using gradient checkpointing can also significantly reduce GPU memory usage, but it may slow down the training speed. |

| MAX_SEQ_LEN |   2048  | The MAX_SEQ_LEN refers to the sequence size limit for a single input image (calculated as H/16 * W/16). A larger value indicates a longer computation sequence and a higher training resolution. The default value I provided is 2048.|



To run the training code, execute the following command.



```bash

export FLUX_FILL_PATH="{path to FLUX.1-Fill-dev}"

python run_train.py  --cfg train_config/ace_plus_lora.yaml

# Training from fft model

export FLUX_FILL_PATH="{path to FLUX.1-Fill-dev}"

export ACE_PLUS_FFT_MODEL="path to ace_plus_fft.safetensors.safetensors"       

python run_train.py  --cfg train_config/ace_plus_fft.yaml 

```



The models trained by ACE++ can be found in ./examples/exp_example/xxxx/checkpoints/xxxx/0_SwiftLoRA/comfyui_model.safetensors.



## 💻 Demo

We have built a GUI demo based on Gradio to help users better utilize the ACE++ model. Just execute the following command.

```bash

export FLUX_FILL_PATH="hf://black-forest-labs/FLUX.1-Fill-dev"

export PORTRAIT_MODEL_PATH="ms://iic/ACE_Plus@portrait/comfyui_portrait_lora64.safetensors"                                                                                                                                      

export SUBJECT_MODEL_PATH="ms://iic/ACE_Plus@subject/comfyui_subject_lora16.safetensors"                                                                                                                                         

export LOCAL_MODEL_PATH="ms://iic/ACE_Plus@local_editing/comfyui_local_lora16.safetensors" 

# Use the model from huggingface

# export PORTRAIT_MODEL_PATH="hf://ali-vilab/ACE_Plus@portrait/comfyui_portrait_lora64.safetensors"        

# export SUBJECT_MODEL_PATH="hf://ali-vilab/ACE_Plus@subject/comfyui_subject_lora16.safetensors"        

# export LOCAL_MODEL_PATH="hf://ali-vilab/ACE_Plus@local_editing/comfyui_local_lora16.safetensors" 

python demo_lora.py

# Use the fft model

export FLUX_FILL_PATH="hf://black-forest-labs/FLUX.1-Fill-dev"

export ACE_PLUS_FFT_MODEL="ms://iic/ACE_Plus@ace_plus_fft.safetensors.safetensors"       

python demo_fft.py

```



## 📚 Limitations

* For certain tasks, such as deleting and adding objects, there are flaws in instruction following. For adding and replacing objects, we recommend trying the repainting method of the local editing model to achieve this.

* The generated results may contain artifacts, especially when it comes to the generation of hands, which still exhibit distortions.



## 📝 Citation

ACE++ is a post-training model based on the FLUX.1-dev series from black-forest-labs. Please adhere to its open-source license. The test materials used in ACE++ come from the internet and are intended for academic research and communication purposes. If the original creators feel uncomfortable, please contact us to have them removed. 



If you use this model in your research, please cite the works of FLUX.1-dev and the following papers:

```bibtex

@article{mao2025ace++,

  title={ACE++: Instruction-Based Image Creation and Editing via Context-Aware Content Filling},

  author={Mao, Chaojie and Zhang, Jingfeng and Pan, Yulin and Jiang, Zeyinzi and Han, Zhen and Liu, Yu and Zhou, Jingren},

  journal={arXiv preprint arXiv:2501.02487},

  year={2025}

}

```

```bibtex

@article{han2024ace,

  title={ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer},

  author={Han, Zhen and Jiang, Zeyinzi and Pan, Yulin and Zhang, Jingfeng and Mao, Chaojie and Xie, Chenwei and Liu, Yu and Zhou, Jingren},

  journal={arXiv preprint arXiv:2410.00086},

  year={2024}

}

```