Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/mikubill/sketch2img

Sketch-to-Image Generation without retraining diffusion models
https://github.com/mikubill/sketch2img

Last synced: about 1 month ago
JSON representation

Sketch-to-Image Generation without retraining diffusion models

Host: GitHub
URL: https://github.com/mikubill/sketch2img
Owner: Mikubill
Created: 2023-01-20T10:35:31.000Z (almost 2 years ago)
Default Branch: main
Last Pushed: 2023-02-01T04:11:30.000Z (almost 2 years ago)
Last Synced: 2024-10-31T00:40:07.216Z (2 months ago)
Language: Python
Homepage:
Size: 424 KB
Stars: 46
Watchers: 5
Forks: 6
Open Issues: 4
Metadata Files:
- Readme: README.md

Awesome Lists containing this project

README

# sketch2img

Sketch-to-Image Generation without retraining diffusion models. (WIP)

Currently supported method:

* Sketch-Guided Text-to-Image Diffusion (google)
* inject with additional pretrained self-attn layer or cross-attn layer (gligen?)

Note: Paint-With-Words already moved to https://github.com/Mikubill/sd-paint-with-words

## Sketch-Guided Text-to-Image Diffusion

[Paper](https://sketch-guided-diffusion.github.io/files/sketch-guided-preprint.pdf) | Demo

![intro](https://sketch-guided-diffusion.github.io/files/scheme_inference.jpg)

Sketch-Guided Text-to-Image is a method proposed by researchers in Google Research to guide the inference process of a pretrained text-to-image diffusion model with an edge predictor that operates on the internal activations of the core network of the diffusion model, encouraging the edge of the synthesized image to follow a reference sketch.

Pretrained LGP Weights: https://huggingface.co/nyanko7/sketch2img-edge-predictor-train/blob/main/edge_predictor.pt