Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/zyxElsa/InST

Official implementation of the paper “Inversion-Based Style Transfer with Diffusion Models” (CVPR 2023)
https://github.com/zyxElsa/InST

Last synced: 3 months ago
JSON representation

Official implementation of the paper “Inversion-Based Style Transfer with Diffusion Models” (CVPR 2023)

Awesome Lists containing this project

README 

        







## Inversion-Based Style Transfer with Diffusion Models



![teaser](./Images/teaser.png)



The artistic style within a painting is the means of expression, which includes not only the painting material, colors, and brushstrokes, but also the high-level attributes including semantic elements, object shapes, etc.  Previous arbitrary example-guided artistic image generation methods often fail to control shape changes or convey elements.  The pre-trained text-to-image synthesis diffusion probabilistic models have achieved remarkable quality, but it often requires extensive textual descriptions to accurately portray attributes of a particular painting. We believe that the uniqueness of an artwork lies precisely in the fact that it cannot be adequately explained with normal language.Our key idea is to learn artistic style directly from a single painting and then guide the synthesis without providing complex textual descriptions.  Specifically, we assume style as a learnable textual description of a painting.  We propose an inversion-based style transfer method (InST), which can efficiently and accurately learn the key information of an image, thus capturing and transferring the complete artistic style of a painting.  We demonstrate the quality and efficiency of our method on numerous paintings of various artists and styles.



For details see the [paper](https://arxiv.org/abs/2211.13203) 



### News

📣📣 Received some requirements, therefore we have added an optional AdaIN module in InST.ipynb to help you achieve results that better meet your needs.



📣📣 See our latest work about attribute-aware image generation with diffusion model [ProSpect: Expanded Conditioning for the Personalization of Attribute-aware Image Generation](https://arxiv.org/abs/2305.16225) in [Code](https://github.com/zyxElsa/ProSpect).



(back to top)



## Getting Started



### Prerequisites



For packages, see environment.yaml.



  ```sh

  conda env create -f environment.yaml

  conda activate ldm

  ```



(back to top)



### Installation



   Clone the repo

   ```sh

   git clone https://github.com/zyxElsa/InST.git

   ```



(back to top)



### Train



   Train InST:

   ```sh

   python main.py --base configs/stable-diffusion/v1-finetune.yaml

               -t 

               --actual_resume ./models/sd/sd-v1-4.ckpt

               -n  

               --gpus 0, 

               --data_root /path/to/directory/with/images

   ```

   

   See `configs/stable-diffusion/v1-finetune.yaml` for more options

   

   Download the pretrained [Stable Diffusion Model](https://huggingface.co/CompVis/stable-diffusion-v-1-4-original/resolve/main/sd-v1-4.ckpt) and save it at ./models/sd/sd-v1-4.ckpt.

   

(back to top)



### Test



   To generate new images, run InST.ipynb

   

(back to top)



### Comparison Data



  For a easier comparison with InST, the results generated with the same _random seed_ `seed=50` and similar _strength_ without any cherry-picking are supplemented. You can download the embedding.pt and content/style/results images [here](https://drive.google.com/drive/folders/1vte8eIp1QG9sQ4iKVeuQnB-RqmqMxVoD?usp=sharing).

  

  ![comparison_data](./Images/comparison_data.png)

  

(back to top)



### Citation

   

   ```sh

   @InProceedings{Zhang_2023_inst,

    author    = {Zhang, Yuxin and Huang, Nisha and Tang, Fan and Huang, Haibin and Ma, Chongyang and Dong, Weiming and Xu, Changsheng},

    title     = {Inversion-Based Style Transfer With Diffusion Models},

    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},

    month     = {June},

    year      = {2023},

    pages     = {10146-10156}

}

   ```

   

(back to top)



## Contact



Please feel free to open an issue or contact us personally if you have questions, need help, or need explanations. Write to one of the following email addresses, and maybe put one other in the cc:



[email protected]



(back to top)



[contributors-shield]: https://img.shields.io/github/contributors/othneildrew/Best-README-Template.svg?style=for-the-badge

[contributors-url]: https://github.com/othneildrew/Best-README-Template/graphs/contributors

[forks-shield]: https://img.shields.io/github/forks/othneildrew/Best-README-Template.svg?style=for-the-badge

[forks-url]: https://github.com/othneildrew/Best-README-Template/network/members

[stars-shield]: https://img.shields.io/github/stars/othneildrew/Best-README-Template.svg?style=for-the-badge

[stars-url]: https://github.com/othneildrew/Best-README-Template/stargazers

[issues-shield]: https://img.shields.io/github/issues/othneildrew/Best-README-Template.svg?style=for-the-badge

[issues-url]: https://github.com/othneildrew/Best-README-Template/issues

[license-shield]: https://img.shields.io/github/license/othneildrew/Best-README-Template.svg?style=for-the-badge

[license-url]: https://github.com/othneildrew/Best-README-Template/blob/master/LICENSE.txt

[linkedin-shield]: https://img.shields.io/badge/-LinkedIn-black.svg?style=for-the-badge&logo=linkedin&colorB=555

[linkedin-url]: https://linkedin.com/in/othneildrew

[product-screenshot]: images/screenshot.png