{"id":20832371,"url":"https://github.com/ermongroup/sdedit","last_synced_at":"2025-05-16T18:08:32.997Z","repository":{"id":41381066,"uuid":"364460992","full_name":"ermongroup/SDEdit","owner":"ermongroup","description":"PyTorch implementation for SDEdit: Image Synthesis and Editing with Stochastic Differential Equations","archived":false,"fork":false,"pushed_at":"2023-02-12T07:40:43.000Z","size":38295,"stargazers_count":1072,"open_issues_count":22,"forks_count":89,"subscribers_count":23,"default_branch":"main","last_synced_at":"2025-04-12T16:59:25.797Z","etag":null,"topics":["controllable-generation","image-editing","image-generation","image-manipulation","pytorch","score-matching"],"latest_commit_sha":null,"homepage":"https://sde-image-editing.github.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ermongroup.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-05-05T04:29:46.000Z","updated_at":"2025-04-10T01:54:00.000Z","dependencies_parsed_at":"2024-01-16T09:13:21.499Z","dependency_job_id":null,"html_url":"https://github.com/ermongroup/SDEdit","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ermongroup%2FSDEdit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ermongroup%2FSDEdit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ermongroup%2FSDEdit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ermongroup%2FSDEdit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ermongroup","download_url":"https://codeload.github.com/ermongroup/SDEdit/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254582907,"owners_count":22095518,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["controllable-generation","image-editing","image-generation","image-manipulation","pytorch","score-matching"],"created_at":"2024-11-18T00:11:34.748Z","updated_at":"2025-05-16T18:08:32.956Z","avatar_url":"https://github.com/ermongroup.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations\n\u003cbr\u003e\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/ermongroup/SDEdit/blob/main/images/sde_animation.gif\" width=\"320\"/\u003e\n\u003c/p\u003e\n\n[**Project**](https://sde-image-editing.github.io/) | [**Paper**](https://arxiv.org/abs/2108.01073) | [**Colab**](https://colab.research.google.com/drive/1KkLS53PndXKQpPlS1iK-k1nRQYmlb4aO?usp=sharing)\n\nPyTorch implementation of **SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations** (ICLR 2022).\n\n[Chenlin Meng](https://cs.stanford.edu/~chenlin/), [Yutong He](http://web.stanford.edu/~kellyyhe/), [Yang Song](https://yang-song.github.io/), [Jiaming Song](http://tsong.me/),\n[Jiajun Wu](https://jiajunwu.com/), [Jun-Yan Zhu](https://www.cs.cmu.edu/~junyanz/), [Stefano Ermon](https://cs.stanford.edu/~ermon/)\n\nStanford and CMU\n\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/ermongroup/SDEdit/blob/main/images/teaser.jpg\" /\u003e\n\u003c/p\u003e\n\nRecently, SDEdit has also been applied to text-guided image editing with large-scale text-to-image models. Notable examples include \u003ca href=\"https://en.wikipedia.org/wiki/Stable_Diffusion\"\u003eStable Diffusion\u003c/a\u003e's img2img function (see  \u003ca href=\"https://github.com/CompVis/stable-diffusion#image-modification-with-stable-diffusion\"\u003ehere\u003c/a\u003e), \u003ca href=\"https://arxiv.org/abs/2112.10741\"\u003eGLIDE\u003c/a\u003e, and \u003ca href=\"https://arxiv.org/abs/2210.03142\"\u003edistilled-SD\u003c/a\u003e. The below example comes from \u003ca href=\"https://arxiv.org/abs/2210.03142\"\u003edistilled-SD\u003c/a\u003e.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/ermongroup/SDEdit/blob/main/images/text_guided_img2img.png\" /\u003e\n\u003c/p\u003e\n\n\n## Overview\nThe key intuition of SDEdit is to \"hijack\" the reverse stochastic process of SDE-based generative models, as illustrated in the figure below. Given an input image for editing, such as a stroke painting or an image with color strokes, we can add a suitable amount of noise to make its artifacts undetectable, while still preserving the overall structure of the image. We then initialize the reverse SDE with this noisy input, and simulate the reverse process to obtain a denoised image of high quality. The final output is realistic while resembling the overall image structure of the input.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/ermongroup/SDEdit/blob/main/images/sde_stroke_generation.jpg\" /\u003e\n\u003c/p\u003e\n\n## Getting Started\nThe code will automatically download pretrained SDE (VP) PyTorch models on\n[CelebA-HQ](https://image-editing-test-12345.s3-us-west-2.amazonaws.com/checkpoints/celeba_hq.ckpt),\n[LSUN bedroom](https://image-editing-test-12345.s3-us-west-2.amazonaws.com/checkpoints/bedroom.ckpt),\nand [LSUN church outdoor](https://image-editing-test-12345.s3-us-west-2.amazonaws.com/checkpoints/church_outdoor.ckpt).\n\n### Data format\nWe save the image and the corresponding mask in an array format ``[image, mask]``, where\n\"image\" is the image with range ``[0,1]`` in the PyTorch tensor format, \"mask\" is the corresponding binary mask (also the PyTorch tensor format) specifying the editing region.\nWe provide a few examples, and ``functions/process_data.py``  will automatically download the examples to the ``colab_demo`` folder.\n\n### Re-training the model\nHere is the [PyTorch implementation](https://github.com/ermongroup/ddim) for training the model.\n\n\n## Stroke-based image generation\nGiven an input stroke painting, our goal is to generate a realistic image that shares the same structure as the input painting.\nSDEdit can synthesize multiple diverse outputs for each input on LSUN bedroom, LSUN church and CelebA-HQ datasets.\n\n\n\nTo generate results on LSUN datasets, please run\n\n```\npython main.py --exp ./runs/ --config bedroom.yml --sample -i images --npy_name lsun_bedroom1 --sample_step 3 --t 500  --ni\n```\n```\npython main.py --exp ./runs/ --config church.yml --sample -i images --npy_name lsun_church --sample_step 3 --t 500  --ni\n```\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/ermongroup/SDEdit/blob/main/images/stroke_based_generation.jpg\" width=\"800\"\u003e\n\u003c/p\u003e\n\n## Stroke-based image editing\nGiven an input image with user strokes, we want to manipulate a natural input image based on the user's edit.\nSDEdit can generate image edits that are both realistic and faithful (to the user edit), while avoid introducing undesired changes.\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/ermongroup/SDEdit/blob/main/images/stroke_edit.jpg\" width=\"800\"\u003e\n\u003c/p\u003e\nTo perform stroke-based image editing, run\n\n```\npython main.py --exp ./runs/  --config church.yml --sample -i images --npy_name lsun_edit --sample_step 3 --t 500  --ni\n```\n\n## Additional results\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/ermongroup/SDEdit/blob/main/images/stroke_generation_extra.jpg\" width=\"800\"\u003e\n\u003c/p\u003e\n\n## References\nIf you find this repository useful for your research, please cite the following work.\n```\n@inproceedings{\n      meng2022sdedit,\n      title={{SDE}dit: Guided Image Synthesis and Editing with Stochastic Differential Equations},\n      author={Chenlin Meng and Yutong He and Yang Song and Jiaming Song and Jiajun Wu and Jun-Yan Zhu and Stefano Ermon},\n      booktitle={International Conference on Learning Representations},\n      year={2022},\n}\n```\n\nThis implementation is based on / inspired by:\n\n- [DDIM PyTorch repo](https://github.com/ermongroup/ddim).\n- [DDPM TensorFlow repo](https://github.com/hojonathanho/diffusion).\n- [PyTorch helper that loads the DDPM model](https://github.com/pesser/pytorch_diffusion).\n- [code structure](https://github.com/ermongroup/ncsnv2).\n\nHere are also some of the interesting follow-up works of SDEdit:\n\n- [Image Modification with Stable Diffusion](https://github.com/CompVis/stable-diffusion#image-modification-with-stable-diffusion)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fermongroup%2Fsdedit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fermongroup%2Fsdedit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fermongroup%2Fsdedit/lists"}