{"id":13488729,"url":"https://github.com/omriav/blended-latent-diffusion","last_synced_at":"2025-03-28T01:37:31.677Z","repository":{"id":41397334,"uuid":"500427876","full_name":"omriav/blended-latent-diffusion","owner":"omriav","description":"Official implementation for \"Blended Latent Diffusion\" [SIGGRAPH 2023]","archived":false,"fork":false,"pushed_at":"2024-06-04T11:38:45.000Z","size":10321,"stargazers_count":565,"open_issues_count":2,"forks_count":34,"subscribers_count":48,"default_branch":"master","last_synced_at":"2024-10-31T00:40:10.895Z","etag":null,"topics":["computer-vision","deep-learning","diffusion","diffusion-models","generative-model","image-generation","multimodal","multimodal-deep-learning","pytorch","text-driven-editing","text-guided-manipulation","text-to-image","text-to-image-synthesis"],"latest_commit_sha":null,"homepage":"https://omriavrahami.com/blended-latent-diffusion-page/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/omriav.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-06-06T12:38:59.000Z","updated_at":"2024-10-28T17:09:13.000Z","dependencies_parsed_at":"2024-10-31T00:32:52.132Z","dependency_job_id":"97f4edc8-982e-4168-a437-18289495370d","html_url":"https://github.com/omriav/blended-latent-diffusion","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/omriav%2Fblended-latent-diffusion","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/omriav%2Fblended-latent-diffusion/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/omriav%2Fblended-latent-diffusion/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/omriav%2Fblended-latent-diffusion/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/omriav","download_url":"https://codeload.github.com/omriav/blended-latent-diffusion/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245952923,"owners_count":20699561,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","deep-learning","diffusion","diffusion-models","generative-model","image-generation","multimodal","multimodal-deep-learning","pytorch","text-driven-editing","text-guided-manipulation","text-to-image","text-to-image-synthesis"],"created_at":"2024-07-31T18:01:20.816Z","updated_at":"2025-03-28T01:37:31.669Z","avatar_url":"https://github.com/omriav.png","language":"Jupyter Notebook","funding_links":[],"categories":["SD-inpaint","Jupyter Notebook"],"sub_categories":[],"readme":"# Blended Latent Diffusion [SIGGRAPH 2023]\n\u003ca href=\"https://omriavrahami.com/blended-latent-diffusion-page/\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=Project\u0026message=Website\u0026color=blue\"\u003e\u003c/a\u003e \n\u003ca href=\"https://arxiv.org/abs/2206.02779\"\u003e\u003cimg src=\"https://img.shields.io/badge/arXiv-2206.02779-b31b1b.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opensource.org/licenses/MIT\"\u003e\u003cimg src=\"https://img.shields.io/badge/License-MIT-yellow.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://pytorch.org/\"\u003e\u003cimg src=\"https://img.shields.io/badge/PyTorch-\u003e=2.1.0-Red?logo=pytorch\"\u003e\u003c/a\u003e\n\n\u003ca href=\"https://omriavrahami.com/blended-latent-diffusion-page/\"\u003e\u003cimg src=\"docs/teaser.png\" /\u003e\u003c/a\u003e\n\n\u003e \u003ca href=\"https://omriavrahami.com/blended-latent-diffusion-page/\"\u003e**Blended Latent Diffusion**\u003c/a\u003e\n\u003e\n\u003e Omri Avrahami, Ohad Fried, Dani Lischinski\n\u003e\n\u003e Abstract: The tremendous progress in neural image generation, coupled with the emergence of seemingly omnipotent vision-language models has finally enabled text-based interfaces for creating and editing images. Handling *generic* images requires a diverse underlying generative model, hence the latest works utilize diffusion models, which were shown to surpass GANs in terms of diversity. One major drawback of diffusion models, however, is their relatively slow inference time. In this paper, we present an accelerated solution to the task of *local* text-driven editing of generic images, where the desired edits are confined to a user-provided mask. Our solution leverages a recent text-to-image Latent Diffusion Model (LDM), which speeds up diffusion by operating in a lower-dimensional latent space. We first convert the LDM into a local image editor by incorporating Blended Diffusion into it. Next we propose an optimization-based solution for the inherent inability of this LDM to accurately reconstruct images. Finally, we address the scenario of performing local edits using thin masks. We evaluate our method against the available baselines both qualitatively and quantitatively and demonstrate that in addition to being faster, our method achieves better precision than the baselines while mitigating some of their artifacts\n\n\u003cdiv\u003e\n  \u003cimg src=\"docs/object_editing.gif\" width=\"200px\"/\u003e\n  \u003cimg src=\"docs/new_object.gif\" width=\"200px\"/\u003e\n  \u003cimg src=\"docs/graffiti.gif\" width=\"200px\"/\u003e\n\u003c/div\u003e\n\n# Applications\n\n### Background Editing\n\u003cimg src=\"docs/applications/background_edit.png\" /\u003e\n\n### Text Generation\n\u003cimg src=\"docs/applications/text.png\" /\u003e\n\n### Multiple Predictions\n\u003cimg src=\"docs/applications/multiple_predictions.png\" /\u003e\n\n### Alter an Existing Object\n\u003cimg src=\"docs/applications/object_edit.png\" /\u003e\n\n### Add a New Object\n\u003cimg src=\"docs/applications/new_object.png\" /\u003e\n\n### Scribble Editing\n\u003cimg src=\"docs/applications/scribble_edit.png\" /\u003e\n\n# Installation\nInstall the conda virtual environment:\n```bash\n$ conda env create -f environment.yaml\n$ conda activate ldm\n```\n\n# Usage\n\n## New :fire: - Stable Diffusion Implementation\nYou can use the newer Stable Diffusion implementation based on [Diffusers](https://github.com/huggingface/diffusers) library.\nFor that, you need to install PyTorch 2.1 and Diffusers via the following commands:\n```bash\n$ conda install pytorch==2.1.0 torchvision==0.16.0  pytorch-cuda=11.8 -c pytorch -c nvidia\n$ pip install -U diffusers==0.19.3\n```\n\n* For using Stable Diffusion XL (requires a stronger GPU), use the following script:\n\n```bash\n$ python scripts/text_editing_SDXL.py --prompt \"a stone\" --init_image \"inputs/img.png\" --mask \"inputs/mask.png\"\n```\nYou can use smaller `--batch_size` in order to save GPU memory.\n\n* For using Stable Diffusion v2.1, use the following script:\n```bash\n$ python scripts/text_editing_SD2.py --prompt \"a stone\" --init_image \"inputs/img.png\" --mask \"inputs/mask.png\"\n```\n\n## Old - Latent Diffusion Model Implementation\nFor using the old implementation, based on the Latent Diffusion Model (LDM), you need first to download the pre-trained weights (5.7GB):\n```bash\n$ mkdir -p models/ldm/text2img-large/\n$ wget -O models/ldm/text2img-large/model.ckpt https://ommer-lab.com/files/latent-diffusion/nitro/txt2img-f8-large/model.ckpt\n```\n\nIf the above link is broken, you can use this [mirror link](https://huggingface.co/omriav/blended-latent-diffusion-ldm/resolve/main/model.ckpt?download=true).\n\nThen, editing the image may require two steps:\n### Step 1 - Generate initial predictions\n```bash\n$ python scripts/text_editing_LDM.py --prompt \"a pink yarn ball\" --init_image \"inputs/img.png\" --mask \"inputs/mask.png\"\n```\n\nThe predictions will be saved in `outputs/edit_results/samples`.\n\nYou can use a larger batch size by specifying `--n_samples` to the maximum number that saturates your GPU.\n\n### Step 2 (optional) - Reconstruct the original background\nIf you want to reconstruct the original image background, you can run the following:\n```bash\n$ python scripts/reconstruct.py --init_image \"inputs/img.png\" --mask \"inputs/mask.png\" --selected_indices 0 1\n```\n\nYou can choose the specific image indices that you want to reconstruct. The results will be saved in `outputs/edit_results/samples/reconstructed_optimization`.\n\n# Citation\nIf you find this project useful for your research, please cite the following:\n```bibtex\n@article{avrahami2023blendedlatent,\n        author = {Avrahami, Omri and Fried, Ohad and Lischinski, Dani},\n        title = {Blended Latent Diffusion},\n        year = {2023},\n        issue_date = {August 2023},\n        publisher = {Association for Computing Machinery},\n        address = {New York, NY, USA},\n        volume = {42},\n        number = {4},\n        issn = {0730-0301},\n        url = {https://doi.org/10.1145/3592450},\n        doi = {10.1145/3592450},\n        journal = {ACM Trans. Graph.},\n        month = {jul},\n        articleno = {149},\n        numpages = {11},\n        keywords = {zero-shot text-driven local image editing}\n}\n\n@InProceedings{Avrahami_2022_CVPR,\n        author    = {Avrahami, Omri and Lischinski, Dani and Fried, Ohad},\n        title     = {Blended Diffusion for Text-Driven Editing of Natural Images},\n        booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},\n        month     = {June},\n        year      = {2022},\n        pages     = {18208-18218}\n}\n```\n\n# Acknowledgements\nThis code is based on [Latent Diffusion Models](https://github.com/CompVis/latent-diffusion).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fomriav%2Fblended-latent-diffusion","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fomriav%2Fblended-latent-diffusion","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fomriav%2Fblended-latent-diffusion/lists"}