https://github.com/dmmaze/sketch2manga
Apply screentone to line drawings or colored illustrations with diffusion models.
https://github.com/dmmaze/sketch2manga
comfyui image-processing manga screentone stable-diffusion
Last synced: 11 months ago
JSON representation
Apply screentone to line drawings or colored illustrations with diffusion models.
- Host: GitHub
- URL: https://github.com/dmmaze/sketch2manga
- Owner: dmMaze
- Created: 2023-12-29T11:00:40.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2025-03-31T03:01:41.000Z (about 1 year ago)
- Last Synced: 2025-03-31T04:19:48.383Z (about 1 year ago)
- Topics: comfyui, image-processing, manga, screentone, stable-diffusion
- Language: Jupyter Notebook
- Homepage:
- Size: 5.96 MB
- Stars: 40
- Watchers: 3
- Forks: 10
- Open Issues: 0
-
Metadata Files:
- Readme: readme.md
Awesome Lists containing this project
README
# Sketch2Manga
[](https://arxiv.org/abs/2403.08266)
Apply screentone to line drawings or colored illustrations with diffusion models.
Sketch2Manga - Drag and drop into ComfyUI to load the workflow
(Source @ini_pmh)
Illustration2Manga - Drag and drop into ComfyUI to load the workflow
(Source @curecu8)
## Usage
### Preperation
### ComfyUI
Download a diffusion model for colorization (this demo used [meinapastel](https://civitai.com/models/11866/meinapastel) for ComfyUI) and [control_v11p_sd15_lineart](https://huggingface.co/lllyasviel/ControlNet-v1-1/blob/main/control_v11p_sd15_lineart.pth).
Download the finetuned [vae](https://huggingface.co/dreMaz/sketch2manga/blob/main/vae/mangatone_default.ckpt) and [diffusion model](https://huggingface.co/dreMaz/sketch2manga/blob/main/mangatone.ckpt) for screening.
Install [ComfyUI](https://github.com/comfyanonymous/ComfyUI).
Clone this repo to the ComfyUI directory and install dependencies:
``` bash
git clone https://github.com/dmMaze/sketch2manga [ComfyUI Directory]/custom_nodes/sketch2manga
cd [ComfyUI Directory]/custom_nodes/sketch2manga
pip install -r requirements.txt
```
Launch ComfyUI, drag and drop the figure above to load the workflow.
### Gradio Demo
Prepare environment
``` bash
conda env create -f conda_env.yaml
pip install git+https://github.com/openai/CLIP.git
```
Download a diffusion model for colorization (this demo used [anything-v4.5](https://huggingface.co/ckpt/anything-v4.5-vae-swapped/tree/main) for sd-webui) and [control_v11p_sd15_lineart](https://huggingface.co/lllyasviel/ControlNet-v1-1/blob/main/control_v11p_sd15_lineart.pth).
Download the finetuned [vae](https://huggingface.co/dreMaz/sketch2manga/blob/main/vae/mangatone_default.ckpt) and [diffusion model](https://huggingface.co/dreMaz/sketch2manga/blob/main/mangatone.ckpt) for screening.
We're using stable-diffusion-webui @ [bef51aed](https://github.com/AUTOMATIC1111/stable-diffusion-webui/commit/bef51aed032c0aaa5cfd80445bc4cf0d85b408b5) and sd-webui-controlnet @ [aa2aa81](https://github.com/Mikubill/sd-webui-controlnet/commit/aa2aa812e86a1f47ef360572888d66027d640f60), other versions might not work. For convient, you can use this [hard fork](https://github.com/dmMaze/stable-diffusion-webui-with-controlnet). Put models metioned above into corresponding sd-webui directories, and launch webui ```python webui.py --api```.
Finally launch the gradio demo:
```
python gradio_demo/launch.py
```
#### SD-WebUI API
There is an example ```webuiapi_demo.ipynb``` showcasing inference using SD-WebUI API, it is a bit outdated though, but the logic applied is the same.
## Comparsion
Our Illustration to Manga method compared with [Mimic Manga](https://lllyasviel.github.io/MangaFilter/) (considered as SOTA)
Illustration (Input)
Mimic Manga
Ours
