Awesome-Controllable-Diffusion

Papers and resources on Controllable Generation using Diffusion Models, including ControlNet, DreamBooth, IP-Adapter.
https://github.com/atfortes/Awesome-Controllable-Diffusion

Last synced: 13 days ago
JSON representation

2023

DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation.
Adding Conditional Control to Text-to-Image Diffusion Models.
Subject-driven Text-to-Image Generation via Apprenticeship Learning.
StyleDrop: Text-to-Image Generation in Any Style.
Face0: Instantaneously Conditioning a Text-to-Image Model on a Face.
Controlling Text-to-Image Diffusion by Orthogonal Finetuning.
Zero-shot spatial layout conditioning for text-to-image diffusion models.
IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models. - adapter.github.io/)] [paper] [code]
ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs.
BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing. - Diffusion-website/)] [paper] [code]
T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models. - Adapter)]
- -Driven-orange?style=flat-square)
GLIGEN: Open-Set Grounded Text-to-Image Generation.
Multi-Concept Customization of Text-to-Image Diffusion. - diffusion/)] [paper] [code]
Regional Prompter
Awesome-LLM-Reasoning
Awesome-Controllable-T2I-Diffusion-Models - to-image diffusion models.

2024

InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning.
Cross-Image Attention for Zero-Shot Appearance Transfer. - image-attention)] [paper] [code]
The Chosen One: Consistent Characters in Text-to-Image Diffusion Models. - chosen-one/)] [paper] [code]
MagicPose: Realistic Human Poses and Facial Expressions Retargeting with Identity-aware Diffusion.
Style Aligned Image Generation via Shared Attention. - aligned-gen.github.io/)] [paper] [code]
Context Diffusion: In-Context Aware Image Generation.
PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding. - maker.github.io/)] [paper] [code]
PALP: Prompt Aligned Personalization of Text-to-Image Models. - aligned.github.io/)] [paper]
InstantID: Zero-shot Identity-Preserving Generation in Seconds.
Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs. - DiffusionMaster)]
UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion. - ptm.github.io/)] [paper]
Kosmos-G: Generating Images in Context with Multimodal Large Language Models - G)]
Training-Free Consistent Text-to-Image Generation.
InstanceDiffusion: Instance-level Control for Image Generation. - xwang/InstanceDiffusion)]
Direct Consistency Optimization for Compositional Text-to-Image Personalization. - t2i.github.io/)] [paper] [code]
RealCompo: Dynamic Equilibrium between Realism and Composition Improves Text-to-Image Diffusion Models.
Visual Style Prompting with Swapping Self-Attention. - ai/Visual-Style-Prompting)]
Gen4Gen: Generative Data Pipeline for Generative Multi-Concept Composition.
Multi-LoRA Composition for Image Generation. - LoRA-Composition/)] [paper] [code]
Continuous Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directions. - control/)] [paper] [code]
- a50b5e?style=flat-square)
Be Yourself: Bounded Attention for Multi-Subject Text-to-Image Generation. - attention/)] [paper] [code]
FlashFace: Human Image Personalization with High-fidelity Identity Preservation. - page/)] [paper] [code]
Concept Weaver: Enabling Multi-Concept Fusion in Text-to-Image Models.
Identity Decoupling for Multi-Subject Personalization of Text-to-Image Models. - t2i.github.io/)] [paper]
MaxFusion: Plug&Play Multi-Modal Generation in Text-to-Image Diffusion Models. - gk.github.io/maxfusion.github.io/)] [paper] [code]
ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback. - ai.github.io/ControlNet_Plus_Plus/)] [paper] [code]
Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model. - adapter.github.io/)] [paper] [code]
MoA: Mixture-of-Attention for Subject-Context Disentanglement in Personalized Image Generation. - research.github.io/mixture-of-attention/)] [paper] [code]
Prompt Optimizer of Text-to-Image Diffusion Models for Abstract Concept Understanding.
StyleBooth: Image Style Editing with Multimodal Instruction. - vilab.github.io/stylebooth-page/)] [paper] [code]
MultiBooth: Towards Generating All Your Concepts in an Image from Text.
MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation. - adapter.github.io/)] [paper] [code]
PuLID: Pure and Lightning ID Customization via Contrastive Alignment.
InstantFamily: Masked Attention for Zero-shot Multi-ID Image Generation.
StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation.
Customizing Text-to-Image Models with a Single Image Pair.
Compositional Text-to-Image Generation with Dense Blob Representations. - 2d.github.io/)] [paper]
FreeCustom: Tuning-Free Customized Image Generation for Multi-Concept Composition. - uofa.github.io/FreeCustom/)] [paper] [code]
RB-Modulation: Training-Free Personalization of Diffusion Models using Stochastic Optimal Control. - modulation.github.io/)] [paper] [code]
IFAdapter: Instance Feature Control for Grounded Text-to-Image Generation. - hah/IFAdapter)]
CSGO: Content-Style Composition in Text-to-Image Generation. - gen.github.io/)] [paper] [code]
Generative Photomontage.
Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from User's Casual Sketches.
- -Driven-orange?style=flat-square)
IPAdapter-Instruct: Resolving Ambiguity in Image-based Conditioning using Instruct Prompts. - research.github.io/IP-Adapter-Instruct.github.io/)] [paper] [code]
ViPer: Visual Personalization of Generative Models via Individual Preference Learning. - VILAB/ViPer)]
Training-free Composite Scene Generation for Layout-to-Image Synthesis. - F/csg)]
SEED-Story: Multimodal Long Story Generation with Large Language Model. - Story)]
Sketch-Guided Scene Image Generation.
Instant 3D Human Avatar Generation using Image Diffusion Models.
Ctrl-X: Controlling Structure and Appearance for Text-To-Image Generation Without Guidance. - x/)] [paper] [code]
Zero-Painter: Training-Free Layout Control for Text-to-Image Synthesis. - AI-Research/Zero-Painter)]
pOps: Photo-Inspired Diffusion Operators.
Personalized Residuals for Concept-Driven Text-to-Image Generation. - residuals/)] [paper]
Make-Your-3D: Fast and Consistent Subject-Driven 3D Content Generation. - Your-3D/)] [paper] [code]
FeedFace: Efficient Inference-based Face Personalization via Diffusion Models. - cd/FeedFace)]
Visual Anagrams: Generating Multi-View Optical Illusions with Diffusion Models.

2025

Bokeh Diffusion: Defocus Blur Control in Text-to-Image Diffusion Models. - diffusion/)] [paper] [code]

Benchmark

SCIENCEQA
ARO
OK-VQA
A-OKVQA - based visual question answering benchmark.
NExT-QA
GQA - world images.
VQA
VQAv2
TAG
Bongard-HOI - object interactions (HOIs) from natural images.
ARC - like form of general fluid intelligence.

Consistency Models

CCM: Adding Conditional Controls to Text-to-Image Consistency Models - blue?style=flat-square)
PIXART-δ: Fast and Controllable Image Generation with Latent Consistency Models - blue?style=flat-square)

Diffusion Models

Star History

![Star History Chart - history.com/#atfortes/Awesome-Controllable-Generation&Timeline)
![Star History Chart - history.com/#atfortes/Awesome-Controllable-Generation&Timeline)
![Star History Chart - history.com/#atfortes/Awesome-Controllable-Generation&Timeline)

Technique