https://github.com/feizc/Gradient-Free-Textual-Inversion

Gradient-Free Textual Inversion for Personalized Text-to-Image Generation
https://github.com/feizc/Gradient-Free-Textual-Inversion

Last synced: 4 months ago
JSON representation

Gradient-Free Textual Inversion for Personalized Text-to-Image Generation

• Host: GitHub
• URL: https://github.com/feizc/Gradient-Free-Textual-Inversion
• Owner: feizc
• Created: 2022-12-13T03:07:26.000Z (about 2 years ago)
• Default Branch: main
• Last Pushed: 2023-01-23T11:45:59.000Z (about 2 years ago)
• Last Synced: 2024-08-01T13:29:01.999Z (7 months ago)
• Language: Python
• Size: 11.4 MB
• Stars: 38
• Watchers: 2
• Forks: 1
• Open Issues: 3
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Gradient-Free Textual Inversion 

Gradient-free textual inversion for personalized text-to-image generation. 

We introduce to use evolution strategy from [OpenAI](https://openai.com/blog/evolution-strategies/) without gradient to optimize the pesudo-word embeddings. 

Our implementation is totally compatible with [diffusers](https://github.com/huggingface/diffusers) and stable diffusion model.



     

     


     

     Evolution process for textual embeddings.

    



## What does this repo do? 

Current personalized text-to-image approaches, which learn to bind a unique identifier with specific subjects or styles in a few given images, usually incorporate a special word and tune its embedding parameters through gradient descent. 

It is natural to question whether we can optimize the textual inversions by only accessing the inference of models?  As only requiring the forward computation to determine the textual inversion retains the benefits of efficient computation and safe deployment. 

Hereto, we introduce a gradient-free framework to optimize the continuous textual inversion in personalized text-to-image generation. 

Specifically, we first initialize the textual inversion with non-parameter cross-attention to ensure the latent embedding space.  

Then, instead of optimizing in the original high-dimensional embedding space, which is intractable for derivative-free optimization, we perform optimization in a decomposition subspace with (i) PCA and (ii) prior normalization through *iterative* evolutionary strategy. 



     

     


     

     Overview of the proposed gradient-free textual inversion framework.

    



## Cases

Some cases generated by standard textual inversion and gradient-free inversion based on stable diffusion model. 



     

     


     

     Cases for the personalized text-to-image generation.

    



## Process

To intialize the textual inversion with cross-attention automatically, run:

```

python initialize_inversion.py 

```

Then, iterative optimize the textual inversion with gradient-free evolution strategy, run:

```

python train_inversion.py 

```

Finally, with the trained textual inversion, you can generated personalized image with ```infer_inversion.py``` script. 

## Acknowledge

This repository is based on [diffusers](https://github.com/huggingface/diffusers) and [textual inversion](https://github.com/huggingface/diffusers/tree/main/examples/textual_inversion) script. Thanks for their clear code.