https://github.com/dmmaze/sketch2manga

Apply screentone to line drawings or colored illustrations with diffusion models.
https://github.com/dmmaze/sketch2manga

comfyui image-processing manga screentone stable-diffusion

Last synced: 3 months ago
JSON representation

Apply screentone to line drawings or colored illustrations with diffusion models.

• Host: GitHub
• URL: https://github.com/dmmaze/sketch2manga
• Owner: dmMaze
• Created: 2023-12-29T11:00:40.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2024-06-30T03:27:55.000Z (6 months ago)
• Last Synced: 2024-10-13T00:59:03.889Z (3 months ago)
• Topics: comfyui, image-processing, manga, screentone, stable-diffusion
• Language: Jupyter Notebook
• Homepage:
• Size: 5.96 MB
• Stars: 35
• Watchers: 3
• Forks: 6
• Open Issues: 0
• Metadata Files:
  • Readme: readme.md

Awesome Lists containing this project

README

        
# Sketch2Manga

[![arXiv](https://img.shields.io/badge/arXiv-2403.08266-)](https://arxiv.org/abs/2403.08266)

Apply screentone to line drawings or colored illustrations with diffusion models.



  

  


    Sketch2Manga - Drag and drop into ComfyUI to load the workflow 

  (Source @ini_pmh)





  

  


    Illustration2Manga - Drag and drop into ComfyUI to load the workflow 

  (Source @curecu8)



## Usage

### Preperation

Install [pytorch](https://pytorch.org/get-started/locally/)  

Install [ComfyUI](https://github.com/comfyanonymous/ComfyUI) or [stable-diffusion-webui](https://github.com/AUTOMATIC1111/stable-diffusion-webui), download a diffusion model for colorization (the demo used [meinapastel](https://civitai.com/models/11866/meinapastel) for ComfyUI, [anything-v4.5](https://huggingface.co/ckpt/anything-v4.5-vae-swapped/tree/main) for sd-webui) and [control_v11p_sd15_lineart](https://huggingface.co/lllyasviel/ControlNet-v1-1/blob/main/control_v11p_sd15_lineart.pth).  

Download the finetuned [vae](https://huggingface.co/dreMaz/sketch2manga/blob/main/vae/mangatone_default.ckpt) and [diffusion model](https://huggingface.co/dreMaz/sketch2manga/blob/main/mangatone.ckpt) for screening.

### ComfyUI

Clone this repo to the ComfyUI directory and install dependencies:

``` bash

git clone https://github.com/dmMaze/sketch2manga [ComfyUI Directory]/custom_nodes/sketch2manga

cd [ComfyUI Directory]/custom_nodes/sketch2manga 

pip install -r requirements.txt

```

Launch ComfyUI, drag and drop the figure above to load the workflow.

### Gradio Demo

Clone this repo and install dependencies, launch sd-webui with argument ```--api```, and run

```

python gradio_demo/launch.py

```

#### SD-WebUI API

There is an example ```webuiapi_demo.ipynb``` showcasing inference using SD-WebUI API, it is a bit outdated though, but the logic applied is the same.

## Comparsion

Our Illustration to Manga method compared with [Mimic Manga](https://lllyasviel.github.io/MangaFilter/) (considered as SOTA)

  

    

      Illustration (Input)

      Mimic Manga

      Ours