{"id":19256713,"url":"https://github.com/markmohr/diffsketchedit","last_synced_at":"2025-04-11T14:11:34.566Z","repository":{"id":243166939,"uuid":"811661505","full_name":"MarkMoHR/DiffSketchEdit","owner":"MarkMoHR","description":"[ICME 2024] Text-based Vector Sketch Editing with Image Editing Diffusion Prior","archived":false,"fork":false,"pushed_at":"2024-06-11T02:07:31.000Z","size":6962,"stargazers_count":10,"open_issues_count":0,"forks_count":4,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-25T10:21:28.555Z","etag":null,"topics":["diffusion-models","image-editing","sketch","sketch-edit","stable-diffusion"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MarkMoHR.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-07T03:43:08.000Z","updated_at":"2025-03-13T07:26:48.000Z","dependencies_parsed_at":"2024-06-11T03:47:02.859Z","dependency_job_id":"820c01e0-d3af-4620-a8d0-95f8e1e71de3","html_url":"https://github.com/MarkMoHR/DiffSketchEdit","commit_stats":null,"previous_names":["markmohr/diffsketchedit"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarkMoHR%2FDiffSketchEdit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarkMoHR%2FDiffSketchEdit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarkMoHR%2FDiffSketchEdit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarkMoHR%2FDiffSketchEdit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MarkMoHR","download_url":"https://codeload.github.com/MarkMoHR/DiffSketchEdit/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248414344,"owners_count":21099459,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["diffusion-models","image-editing","sketch","sketch-edit","stable-diffusion"],"created_at":"2024-11-09T19:06:43.406Z","updated_at":"2025-04-11T14:11:34.541Z","avatar_url":"https://github.com/MarkMoHR.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Text-based Vector Sketch Editing with Image Editing Diffusion Prior (ICME 2024)\n\n[[Paper]](https://www.sysu-imsl.com/files/ICME2024/ICME2024_sketch_editing_final.pdf)\n\nThis code is used for editing vector sketches with text prompts.\n\n\u003cimg src='docs/figures/teaser3.gif'\u003e\n\n## Outline\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Citation](#citation)\n\n## Installation\n\n1. Please follow instructions in [ximinng/DiffSketcher](https://github.com/ximinng/DiffSketcher?tab=readme-ov-file#step-by-step) for the step-by-step environment preparation.\n2. Download the [CompVis/stable-diffusion-v1-4](https://huggingface.co/CompVis/stable-diffusion-v1-4/tree/main) models and place them somewhere. Follow file structure [here](https://github.com/MarkMoHR/DiffSketchEdit/tree/main/StableDiffusionModels/CompVis/stable-diffusion-v1-4).\n3. Finally, modify the directory path of your downloaded models to `huggingface_model_dict[\"sd14\"]`([Line 11](https://github.com/MarkMoHR/DiffSketchEdit/blob/main/methods/diffusers_warp/__init__.py#L11)) of `./methods/diffusers_warp/__init__.py`.\n\n## Quick Start\n\nUse the code `run_painterly_render.py` and scroll to [Line 81](https://github.com/MarkMoHR/DiffSketchEdit/blob/main/run_painterly_render.py#L81). Then, modify the code according to the following instructions:\n\n1. Set one or more seeds, or choose random ones.\n2. Choose the editing type. `replace`, `refine` and `reweight` stand for editing modes Word Swap, Prompt Refinement and Attention Re-weighting, respectively.\n3. Set the prompt information.\n\n### Examples\n\n(a) Word Swap (`replace`)\n\n```\nseeds_list = [25760]\nargs.edit_type = \"replace\"\n\nPromptInfo(prompts=[\"A painting of a squirrel eating a burger\",\n                    \"A painting of a rabbit eating a burger\",\n                    \"A painting of a rabbit eating a pumpkin\",\n                    \"A painting of a owl eating a pumpkin\"],\n           token_ind=5,\n           changing_region_words=[[\"\", \"\"], [\"squirrel\", \"rabbit\"], [\"burger\", \"pumpkin\"], [\"rabbit\", \"owl\"]])\n```\n\n- `token_ind`: indicate the index of cross-attn maps for initializing strokes.\n- `changing_region_words`: for local editing. Type in two words to indicate the changing regions during each edit. Use empty strings for the first edit.\n\n\n| Original image and sketch | Edited image and sketch 1 | Edited image and sketch 2 | Edited image and sketch 3 | \n|:-------------:|:-------------------:|:----------------------:|:--------:|\n| \u003cimg src=\"docs/figures/replace/ldm_generated_image0.png\" style=\"width: 200px\"\u003e | \u003cimg src=\"docs/figures/replace/ldm_generated_image1.png\" style=\"width: 200px\"\u003e | \u003cimg src=\"docs/figures/replace/ldm_generated_image2.png\" style=\"width: 200px\"\u003e | \u003cimg src=\"docs/figures/replace/ldm_generated_image3.png\" style=\"width: 200px\"\u003e |\n| \u003cimg src=\"docs/figures/replace/visual_best-rendered0.png\" style=\"width: 200px\"\u003e | \u003cimg src=\"docs/figures/replace/visual_best-rendered1.png\" style=\"width: 200px\"\u003e | \u003cimg src=\"docs/figures/replace/visual_best-rendered2.png\" style=\"width: 200px\"\u003e | \u003cimg src=\"docs/figures/replace/visual_best-rendered3.png\" style=\"width: 200px\"\u003e |\n\n(b) Prompt Refinement (`refine`)\n\n```\nseeds_list = [53487]\nargs.edit_type = \"refine\"\n\nPromptInfo(prompts=[\"An evening dress\",\n                    \"An evening dress with sleeves\",\n                    \"An evening dress with sleeves and a belt\"],\n           token_ind=3,\n           changing_region_words=[[\"\", \"\"], [\"\", \"sleeves\"], [\"\", \"belt\"]]),\n```\n\n- `changing_region_words`: set an empty string for the first words.\n\n| Original image and sketch | Edited image and sketch 1 | Edited image and sketch 2 |\n|:-------------:|:-------------------:|:----------------------:|\n| \u003cimg src=\"docs/figures/refine/ldm_generated_image0.png\" style=\"width: 200px\"\u003e | \u003cimg src=\"docs/figures/refine/ldm_generated_image1.png\" style=\"width: 200px\"\u003e | \u003cimg src=\"docs/figures/refine/ldm_generated_image2.png\" style=\"width: 200px\"\u003e |\n| \u003cimg src=\"docs/figures/refine/visual_best-rendered0.png\" style=\"width: 200px\"\u003e | \u003cimg src=\"docs/figures/refine/visual_best-rendered1.png\" style=\"width: 200px\"\u003e | \u003cimg src=\"docs/figures/refine/visual_best-rendered2.png\" style=\"width: 200px\"\u003e |\n\n\n(c) Attention Re-weighting (`reweight`)\n\n```\nseeds_list = [35491]\nargs.edit_type = \"reweight\"\n\nPromptInfo(prompts=[\"An emoji face with moustache and smile\"] * 3,\n           token_ind=3,\n           changing_region_words=[[\"\", \"\"], [\"moustache\", \"moustache\"], [\"smile\", \"smile\"]],\n           reweight_word=[\"moustache\", \"smile\"],\n           reweight_weight=[-1.0, 3.0]),\n```\n\n- `changing_region_words`: set the same words for each pair.\n- `reweight_word` / `reweight_weight`: word or weight for reweighting at each edit.\n\n\n| Original image and sketch | Edited image and sketch 1 | Edited image and sketch 2 |\n|:-------------:|:-------------------:|:----------------------:|\n| \u003cimg src=\"docs/figures/reweight/ldm_generated_image0.png\" style=\"width: 200px\"\u003e | \u003cimg src=\"docs/figures/reweight/ldm_generated_image1.png\" style=\"width: 200px\"\u003e | \u003cimg src=\"docs/figures/reweight/ldm_generated_image2.png\" style=\"width: 200px\"\u003e |\n| \u003cimg src=\"docs/figures/reweight/visual_best-rendered0.png\" style=\"width: 200px\"\u003e | \u003cimg src=\"docs/figures/reweight/visual_best-rendered1.png\" style=\"width: 200px\"\u003e | \u003cimg src=\"docs/figures/reweight/visual_best-rendered2.png\" style=\"width: 200px\"\u003e |\n\n\n## Acknowledgement\n\nThe project is built upon [ximinng/DiffSketcher](https://github.com/ximinng/DiffSketcher) and [google/prompt-to-prompt](https://github.com/google/prompt-to-prompt). We thank all the authors for their effort.\n\n## Citation\n\nIf you use the code please cite:\n\n```\n@inproceedings{mo2024text,\n  title={Text-based Vector Sketch Editing with Image Editing Diffusion Prior},\n  author={Mo, Haoran and Lin, Xusheng and Gao, Chengying and Wang, Ruomei},\n  booktitle={2024 IEEE International Conference on Multimedia and Expo (ICME)},\n  pages={1--6},\n  year={2024},\n  organization={IEEE}\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarkmohr%2Fdiffsketchedit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarkmohr%2Fdiffsketchedit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarkmohr%2Fdiffsketchedit/lists"}