{"id":13488762,"url":"https://github.com/kangyeolk/Paint-by-Sketch","last_synced_at":"2025-03-28T01:37:45.587Z","repository":{"id":64722167,"uuid":"577753729","full_name":"kangyeolk/Paint-by-Sketch","owner":"kangyeolk","description":"Stable Diffusion-based image manipulation method with a sketch and reference image","archived":false,"fork":false,"pushed_at":"2023-04-23T12:45:25.000Z","size":31593,"stargazers_count":172,"open_issues_count":4,"forks_count":9,"subscribers_count":8,"default_branch":"main","last_synced_at":"2024-08-01T18:40:04.526Z","etag":null,"topics":["computer-vision","diffusion-models","image-editing","image-generation","image-manipulation","pytorch","stable-diffusion"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kangyeolk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-12-13T12:57:10.000Z","updated_at":"2024-07-03T08:32:58.000Z","dependencies_parsed_at":"2024-01-18T22:48:54.627Z","dependency_job_id":null,"html_url":"https://github.com/kangyeolk/Paint-by-Sketch","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kangyeolk%2FPaint-by-Sketch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kangyeolk%2FPaint-by-Sketch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kangyeolk%2FPaint-by-Sketch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kangyeolk%2FPaint-by-Sketch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kangyeolk","download_url":"https://codeload.github.com/kangyeolk/Paint-by-Sketch/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222333976,"owners_count":16968058,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","diffusion-models","image-editing","image-generation","image-manipulation","pytorch","stable-diffusion"],"created_at":"2024-07-31T18:01:21.382Z","updated_at":"2025-03-28T01:37:45.580Z","avatar_url":"https://github.com/kangyeolk.png","language":"Python","funding_links":[],"categories":["SD-inpaint"],"sub_categories":[],"readme":"# Paint-by-Sketch\n\n### [Paper](https://arxiv.org/abs/2304.09748)\n\u003c!-- \u003cbr\u003e --\u003e\n[Kangyeol Kim](https://kangyeolk.github.io/), [Sunghyun Park](https://psh01087.github.io/), [Junsoo Lee](https://ssuhan.github.io/) and [Jaegul Choo](https://sites.google.com/site/jaegulchoo/?pli=1).\n\u003c!-- \u003cbr\u003e --\u003e\n\n### Teaser\n![Teaser](asset/teaser.png)\n\n### Multi-backgrounds\n![Multi-backgrounds](asset/multi-backgrounds.png)\n\n### Multi-references\n![Multi-references](asset/multi-references.png)\n\n## Abstract\n\u003eRecent remarkable improvements in large-scale text-to-image generative models have shown promising results in generating high-fidelity images. To further enhance editability and enable fine-grained generation, we introduce a multi-input-conditioned image composition model that incorporates a sketch as a novel modal, alongside a reference image. Thanks to the edge-level controllability using sketches, our method enables a user to edit or complete an image sub-part with a desired structure (i.e., sketch) and content (i.e., reference image). Our framework fine-tunes a pre-trained diffusion model to complete missing regions using the reference image while maintaining sketch guidance. Albeit simple, this leads to wide opportunities to fulfill user needs for obtaining the in-demand images.Through extensive experiments, we demonstrate that our proposed method offers unique use cases for image manipulation, enabling user-driven modifications of arbitrary scenes.\n\u003e\n\n## Environment \u0026 Pre-trained models\n\n### Dependancies\n\n```\n$ conda env create -f environment.yaml\n$ conda activate paint_sketch\n$ pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/cu113\n$ pip install opencv-python==4.6.0.66 opencv-python-headless==4.6.0.66 matplotlib==3.2.2 streamlit==1.14.1 streamlit-drawable-canvas==0.9.2\n$ pip install git+https://github.com/openai/CLIP.git\n```\n\n### Download checkpoints\n* [Google Drive](https://drive.google.com/file/d/1jCDamfsknQj-A28RMA6Blws8iQn0y1ez/view?usp=share_link)\n* Place a downloaded file as below:\n```\nPaint-by-Sketch\n    pretrained_models/\n        model-modified-12channel.ckpt        \n    models/\n        Cartoon_v1_aesthetic/\n            ...\n    ...\n```\n\n## Data preparation\n\n* Sketch extraction\n```bash \nbash preprocess_dataset/run_preprocess.sh \u003cpath/to/image_root\u003e \u003cgpu_id\u003e\n# e.g., \n# bash preprocess_dataset/run_preprocess.sh /home/nas2_userF/kangyeol/Project/webtoon2022/Paint-by-Sketch/samples 7\n```\n\n* Result\n```\nIMAGE_ROOT\n    images/\n        000000.png\n        000001.png\n        ...\n    sketch_bin/\n        000000.png\n        000001.png\n        ...\n    sketch(Not used)/\n        ... \n    ...\n```\n\n\n## Training\n\n```bash\nbash cartoon_train.sh \u003cgpu_ids\u003e \u003cpath/to/logdir\u003e \u003cpath/to/config\u003e\n\n# e.g,\n# bash cartoon_train.sh 0,1 models/test configs/v1_aesthetic_sketch_image.yaml\n```\n\n* You need to match the number of `gpu_ids` and the number of gpus in the `lightning.trainer.yaml` (`gpu_ids`=2,3 with '0,1' in config file).\n\n## Demo\n\n0. Running `streamlit` server \n```bash\nstreamlit run demo/app.py --server.port=8507 --server.fileWatcherType none\n```\n\n1. Upload the source image\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"asset/1_load_image.png\"\u003e\n\u003c/p\u003e\n\n2. Draw mask and sketch separately\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"asset/2_draw_mask_sketch.png\"\u003e\n\u003c/p\u003e\n\n* The 1st and 2nd canvases are panels where you can draw masks and sketches.\u003cbr\u003e\n* In the 3rd canvas, you can view the drawn mask and sketch overlaid together.\u003cbr\u003e\n\n\n3. Upload a reference image.\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"asset/3_load_exemplar.png\"\u003e\n\u003c/p\u003e\n\n* Select a image in the left panel.\u003cbr\u003e\n* Click `Read Exemplar` button.\u003cbr\u003e\n* Crop the image partially with bounding box.\u003cbr\u003e\n\n4. Inference and export\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"asset/4_inference_export.png\"\u003e\n\u003c/p\u003e\n\n* Perform inference with the drawn mask, sketch, and the cropped image as conditions. \u003cbr\u003e\n* You can adjust the `scale` and `sketch strength` in the left panel.\u003cbr\u003e\n* ou can save images in grid format through `Export` button.\u003cbr\u003e\n\n## Issues\n\n* If the screen size is not large enough and the canvas size changes, there will be misalignment in the drawn mask and sketch.\n\n## Citation\n```\n@misc{kim2023referencebased,\n    title={Reference-based Image Composition with Sketch via Structure-aware Diffusion Model},\n    author={Kangyeol Kim and Sunghyun Park and Junsoo Lee and Jaegul Choo},\n    year={2023},\n    eprint={2304.09748},\n    archivePrefix={arXiv},\n    primaryClass={cs.CV}\n}\n```\n\n## License\nThe code in this repository is released under the MIT License.\n\n## Acknowledges\nThis code borrows heavily from [Stable Diffusion](https://github.com/CompVis/stable-diffusion) and [Paint-by-Example](https://github.com/Fantasy-Studio/Paint-by-Example).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkangyeolk%2FPaint-by-Sketch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkangyeolk%2FPaint-by-Sketch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkangyeolk%2FPaint-by-Sketch/lists"}