https://github.com/CHAITron/sketchdeco-code
Official implementation of "SketchDeco: Decorating B&W Sketches with Colour"
https://github.com/CHAITron/sketchdeco-code
Last synced: about 1 month ago
JSON representation
Official implementation of "SketchDeco: Decorating B&W Sketches with Colour"
- Host: GitHub
- URL: https://github.com/CHAITron/sketchdeco-code
- Owner: CHAITron
- License: mit
- Created: 2024-05-31T00:01:11.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2024-12-24T20:37:03.000Z (6 months ago)
- Last Synced: 2024-12-24T21:24:26.253Z (6 months ago)
- Language: Python
- Size: 20.3 MB
- Stars: 55
- Watchers: 3
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-llm-projects - SketchDeco - WeiΓ-Skizzen mit Farbe. (Projekte / π Image)
README
# SketchDeco
Official repo for [SketchDeco: Decorating B&W Sketches with Colour](https://arxiv.org/abs/2405.18716).See [Project Page](https://chaitron.github.io/SketchDeco/) for more examples.


SketchDeco is a training-free sketch colourisation approach that turns free-hand sketches, masks, and colour palettes into realistic images without user-defined text prompts.
## Prerequisites
Our codebase is tested on Nvidia RTX 4090 Super GPUs and Google Colab A100 instance. A VRAM of 23 GB is recommended, though this may vary depending on the input samples (minimum 20 GB).
### Install Dependencies
```
git clone https://github.com/CHAITron/sketchdeco-code.git
cd sketchdeco-code
pip install -r requirements.txt
```
### Download Model (other models will automatically download)```
mkdir ckpt
wget -O ckpt/sd-v2-1_512-ema-pruned.ckpt https://huggingface.co/stabilityai/stable-diffusion-2-1-base/resolve/main/v2-1_512-ema-pruned.ckpt?download=true
```## Sketch, Masks, and Palettes Preparation
We provided two examples in inputs/ folder. In each example, sketch.png is an input sketch with two colourised Region-of-Interests (ROIs) (mask1+palette1 and mask2+palette2).In order to add more ROIs, user can add mask3+palette3, ... . The input data structure is like this:
```
inputs
βββ example1
β βββ sketch.xxx
β βββ mask1.xxx
β βββ palette1.xxx
β βββ mask2.xxx
β βββ palette2.xxx
β βββ ...
βββ example2
β βββ sketch.xxx
β βββ mask1.xxx
β βββ palette1.xxx
β βββ mask2.xxx
β βββ palette2.xxx
β βββ ...
βββ ...
```### Sketch Preparation
Input Sketch can be any B&W sketch with size 512x512. In the examples, we got them from https://www.freepik.com/:
![]()
### Mask and Palette Preparation
Since, mask#n and palette#n are linked together, user is expected to add two files (mask+palette) when adding new RoI. For mask, it is binary image shared same resolution with input sketch (512x512). In our examples, we used **Adobe Photoshop** to create masks using the **Brush** and **Magic Wand** tools. Here are some illustrations of the masks:
![]()
Colour palette can be described in text file. Each line should contain one colour hex-code:
```
#DC143C
#8B0000
#000000
```## Running SketchDeco
To execute the SketchDeco, run the following commands:
```
!python main.py --indir ./inputs \
--outdir ./outputs \
--tau 0.4 \
--seed 1284
```
P.S. seed is just used for reproducing the following results from inputs/:
Tau value is a scaling factor representing a trade-off between harmonisation and faithfulness. A low Ο = 0.0 gives better harmonisation of colours but lacks faithfulness (e.g., missing regions in the wings of βbutterflyβ) whereas a high Ο = 1.0 gives highly faithful image generation but lacks sufficient harmonisation of colours (e.g., the red patch on βcowβ).

## TODO
- [x] Release inference code.
- [ ] Release GUI version.## Acknowledgments
Our work is built on the amazing achievements of those who came before us. A huge thank you to the brilliant minds who created the foundation for our code:
[TF-ICON](https://github.com/Shilin-LU/TF-ICON), [Stable-Diffusion](https://github.com/Stability-AI/stablediffusion) and [Prompt-to-Prompt](https://github.com/google/prompt-to-prompt).## Citation
If you find the repo useful, please consider citing:
```
@article{utintu2024sketchdeco,
author = {Chaitat Utintu and Pinaki Nath Chowdhury and Aneeshan Sain and Subhadeep Koley and Ayan Kumar Bhunia and Yi-Zhe Song},
title = {SketchDeco: Decorating B&W Sketches with Colour},
journal = {arXiv preprint arXiv:2405.18716},
year = {2024},
}
```