https://github.com/bytedance/ComfyUI-HyperLoRA
Official implementation in ComfyUI of CVPR 2025 paper "HyperLoRA: Parameter-Efficient Adaptive Generation for Portrait Synthesis"
https://github.com/bytedance/ComfyUI-HyperLoRA
Last synced: about 2 months ago
JSON representation
Official implementation in ComfyUI of CVPR 2025 paper "HyperLoRA: Parameter-Efficient Adaptive Generation for Portrait Synthesis"
- Host: GitHub
- URL: https://github.com/bytedance/ComfyUI-HyperLoRA
- Owner: bytedance
- License: gpl-3.0
- Created: 2025-04-17T07:55:26.000Z (about 2 months ago)
- Default Branch: main
- Last Pushed: 2025-04-23T09:22:54.000Z (about 2 months ago)
- Last Synced: 2025-04-23T10:29:20.620Z (about 2 months ago)
- Language: Python
- Homepage:
- Size: 4.7 MB
- Stars: 18
- Watchers: 2
- Forks: 1
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-comfyui - **ComfyUI-HyperLoRA** - Efficient Adaptive Generation for Portrait Synthesis' (All Workflows Sorted by GitHub Stars)
README
#
HyperLoRA: Parameter-Efficient Adaptive Generation for Portrait Synthesis
CVPR 2025 (Highlight)
Mengtian Li*,
Jinshu Chen*,
Wanquan Feng*✉,
Bingchuan Li,
Fei Dai,
Songtao Zhao,
Qian He
Intelligent Creation, ByteDance
## Abstract
Personalized portrait synthesis, essential in domains like social entertainment, has recently made significant progress. Person-wise fine-tuning based methods, such as LoRA and DreamBooth, can produce photorealistic outputs but need training on individual samples, consuming time and resources and posing an unstable risk. Adapter based techniques such as IP-Adapter freeze the foundational model parameters and employ a plug-in architecture to enable zero-shot inference, but they often exhibit a lack of naturalness and authenticity, which are not to be overlooked in portrait synthesis tasks. In this paper, we introduce a parameter-efficient adaptive generation method, namely HyperLoRA, that uses an adaptive plug-in network to generate LoRA weights, merging the superior performance of LoRA with the zero-shot capability of adapter scheme. Through our carefully designed network structure and training strategy, we achieve zero-shot personalized portrait generation (supporting both single and multiple image inputs) with high photorealism, fidelity, and editability.
## Overview
We explicitly decompose the HyperLoRA into a Hyper ID-LoRA and a Hyper Base-LoRA. The former is designed to learn ID information while the latter is expected to fit others, e.g. background and clothing. Such a design helps to prevent irreverent features leaking to ID-LoRA. During the training, we fix the weights of the pretrained SDXL base model and encoders, allowing only HyperLoRA modules updated by Backpropagation. At the inference stage, the Hyper ID-LoRA integrated into SDXL generates personalized images while the Hyper Base-LoRA is optional.
## Usage
### Create Model Folders
Navigate to your ComfyUI's ```models``` folder and create the following subfolders:
```
models/
├── hyper_lora/
│ ├── clip_processor/ # Directory for CLIP processor files
│ ├── clip_vit/ # Directory for CLIP ViT model files
│ └── hyper_lora/ # Directory for HyperLoRA model files
└── insightface/
└── models/ # Directory for InsightFace model files
```
### Download Models
First, download the [CLIP processor](https://huggingface.co/openai/clip-vit-large-patch14/raw/main/preprocessor_config.json) to ```models/hyper_lora/clip_processor/clip_vit_large_14_processor```
Second, download the [CLIP ViT model](https://huggingface.co/openai/clip-vit-large-patch14/resolve/main/model.safetensors) and [config](https://huggingface.co/openai/clip-vit-large-patch14/raw/main/config.json) to ```models/hyper_lora/clip_vit/clip_vit_large_14```
Then, download the [antelopev2](https://github.com/deepinsight/insightface/tree/master/python-package#model-zoo) and unzip it to ```models/insightface/models```
Finally, download the [HyperLoRA model](https://huggingface.co/bytedance-research/HyperLoRA) to ```models/hyper_lora/hyper_lora```
We have released two versions of HyperLoRA. sdxl_hyper_id_lora_v1_fidelity offers better fidelity, while sdxl_hyper_id_lora_v1_edit provides better editability. Note that we trained for some additional steps using FP16 precision and distilled the 0.25x Base LoRA into the ID LoRA, which further reduced the model size and the GPU memory usage.
### Example Workflows
The workflows depend on [ComfyUI_ADV_CLIP_emb](https://github.com/BlenderNeko/ComfyUI_ADV_CLIP_emb) and [ComfyUI-Impact-Pack](https://github.com/ltdrdata/ComfyUI-Impact-Pack) plugins.
Name
Workflow
Result
T2I
HyperLoRA-T2I.json
T2I + FaceDetailer
HyperLoRA-T2I-FaceDetailer.json
T2I + ControlNet
HyperLoRA-T2I-ControlNet.json
T2I + InstantID
HyperLoRA-T2I-InstantID.json
We fixed the random seed in all workflows so that the results can be fully reproduced. All the input images required by these workflows can be found in the ```assets``` folder.
### Tips
**IMPORTANT:**
- Please confirm that all model and config files are placed in correct directories.
- The text prompt should start with the LoRA trigger words `fcsks fxhks fhyks, ` to enable LoRA.
- It is recommended to set `stop_at_clip_layer` to `-2` to align with our training config.
HyperLoRA is not compatible with all SDXL base models. We have tested that it is compatible with [LEOSAM's HelloWorld XL 3.0](https://civitai.com/models/43977?modelVersionId=262973), [CyberRealistic XL v1.1](https://civitai.com/models/312530?modelVersionId=395041), and [RealVisXL v4.0](https://civitai.com/models/139562?modelVersionId=344487), but incompatible with [ArienMixXL v4.0](https://civitai.com/models/118913?modelVersionId=322265). Among them, RealVisXL v4.0 may achieve the best results.
LEOSAM 😊
CyberRealistic 😊
RealVis 😊
ArienMix 😭
The LoRA weight are usually around 0.75 to 0.85. If the prompt is very long, you can appropriately increase the weight (as well as the weight of trigger words). If better editability is required, you can appropriately decrease the weight.
Sunglasses
Baby
Sketch
Sand Sculpture
Due to the limited resolution of the trained faces, it is recommended to use a FaceDetailer plugin to repair small faces or use ControlNet to increase stability. In addition, combining HyperLoRA with the ControlNet of InstantID can usually further improve the facial similarity.
Furthermore, HyperLoRA can also provide a good initialization for LoRA training. You can obtain a better ID LoRA with only about 50 steps of finetuning.
## Disclaimer and Licenses
Our code is under the GPL 3.0 license. You can use, modify, and distribute it, with derivative works also under GPL 3.0. Our models follow the CC BY-NC 4.0 license, allowing non-commercial sharing and adaptation with proper credit. Third-party models like SDXL, CLIP, and InsightFace are subject to their original licenses.
This project aims to advance the field of generative AI. We disclaim any liability for misuse or unauthorized use of the code, models, or related materials. You must comply with all relevant licenses and laws when using this plugin.
## Citation
If you find HyperLoRA useful in your research, please kindly cite our paper:
```bibtex
@article{li2025hyperlora,
title={HyperLoRA: Parameter-Efficient Adaptive Generation for Portrait Synthesis},
author={Mengtian Li and Jinshu Chen and Wanquan Feng and Bingchuan Li and Fei Dai and Songtao Zhao and Qian He},
journal={arXiv preprint arXiv:2503.16944},
year={2025}
}
```