{"id":13778869,"url":"https://github.com/FilippoMB/Diffusion_models_tutorial","last_synced_at":"2025-05-11T12:32:11.254Z","repository":{"id":112116179,"uuid":"529918328","full_name":"FilippoMB/Diffusion_models_tutorial","owner":"FilippoMB","description":null,"archived":false,"fork":false,"pushed_at":"2025-03-10T16:20:31.000Z","size":61743,"stargazers_count":97,"open_issues_count":0,"forks_count":9,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-10T17:31:34.217Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FilippoMB.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-28T16:33:41.000Z","updated_at":"2025-03-10T16:20:36.000Z","dependencies_parsed_at":null,"dependency_job_id":"64c1b6c9-29ff-456b-a55f-14546769ed70","html_url":"https://github.com/FilippoMB/Diffusion_models_tutorial","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FilippoMB%2FDiffusion_models_tutorial","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FilippoMB%2FDiffusion_models_tutorial/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FilippoMB%2FDiffusion_models_tutorial/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FilippoMB%2FDiffusion_models_tutorial/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FilippoMB","download_url":"https://codeload.github.com/FilippoMB/Diffusion_models_tutorial/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253566956,"owners_count":21928758,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T18:00:58.386Z","updated_at":"2025-05-11T12:32:11.222Z","avatar_url":"https://github.com/FilippoMB.png","language":"Jupyter Notebook","readme":"# Diffusion models tutorial \n\nA while ago [Stable Diffusion Public Release](https://stability.ai/blog/stable-diffusion-public-release) made available to everyone one of the most powerful deep learning models for image generation. \n\nThis repository contains:\n\n- Tutorials on the theory behind diffusion models and the software libraries used to implement them.\n- A step-by-step guide on how to implement a diffusion model from scratch.\n- A collection of scripts and notebooks that can be used to generate images with Stable Diffusion.\n- A basic guide to prompt engineering.\n- A list of resources to dig deeper into the world of diffusion models.\n\n## 1. 🚀 Quick start\n\n### 🧱 Learn the basics\n\nCheck these [slides](https://docs.google.com/presentation/d/1jUO9jZLtUGoK7kgg0kurBgDwDsNOLybrYKU-O2y98xM/edit?usp=sharing) for a short introduction about the basic idea behind diffusion models and stable diffusion.\n\n### 💻 Play with notebooks\n\n\u003e [!WARNING] \n\u003e Since the libraries used in the notebooks below are often updated, the notebooks might stop working at some point. Please, open an issue if you encounter problems and bugs.\n\nTry out **Stable Diffusion** by running one of the Colab notebooks below.\n\n- Text to image [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1MyESLsR8D5l_EBqumwxL0eMzNmd3uqs6?usp=sharing)\n- Image to image [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1zjr9n60q3G8Qd87WG3ZnjDgI-D1YBCbG?usp=sharing)\n\nTo try out **Stable Diffusion 2**, run one of the Colab notebooks below.\n- Text to image [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1e8MV75aD66It0WXo5-ddUEjfvhVG8URv?usp=sharing)\n- Impainting [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1JuTzAY0NzojvSU-rB4BNI9F8z3tcv9iX?usp=sharing)\n- Super-resolution [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1qnKCWs4IB-n1xfCjeNgbaSu-mFwp7bLv?usp=sharing)\n- Depth-to-image [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1dS9E8lBJywuoCmZiZAVXo6IaGw4Te5B0?usp=sharing)\n\n### 🤓 Understand the theory and learn to build pipelines\n\n- Understand the theory behind stable diffusion models and learn how to code a simple diffusion model from scratch in [this notebook](https://nbviewer.org/github/FilippoMB/Diffusion_models_tutorial/blob/main/diffusion_from_scratch.ipynb).\n- Become familiar with the stable diffusion pipeline and the diffusers 🧨 library  in [this notebook](https://nbviewer.org/github/FilippoMB/Diffusion_models_tutorial/blob/main/Diffusers_library.ipynb).\n\nTo run the notebooks you need to have several libraries installed. You can do that by installing Anaconda (or [Miniconda](https://docs.anaconda.com/free/miniconda/miniconda-install/)) and then creating the environment using the provided env files.\n\nFirst, try to create the environment using [environment.yml](https://github.com/FilippoMB/Diffusion_models_tutorial/blob/main/environment.yml):\n\n````bash\nconda env create -f environment.yml\n````\n\nIf that doesn't work, try with [env_flex.yml](https://github.com/FilippoMB/Diffusion_models_tutorial/blob/main/env_flex.yml) that allows for a more flexible installation. \n\n````bash\nconda env create -f env_flex.yml\n````\n\nThe risk here is that it will install more recent versions of the software packages and the notebooks might give some errors.\nYou might need to this more flexible install also if you are on Windows.\nIf you are on MacOS, instead, use [env_mac.yml](https://github.com/FilippoMB/Diffusion_models_tutorial/blob/main/env_mac.yml):\n\n````bash\nconda env create -f env_flex.yml\n````\n\n## 2. 👷🏻 Prompt engineering guide\nLet's say you want to draw an image of ``lion``. The raw prompt, ``lion`` will give you images that are usually a bit chaotic or of worse quality.\n\n\n\u003cdiv\u003e\n\u003cimg src=\"./img/lion_short.png\" width=\"250\"/\u003e\n\u003c/div\u003e\n\nTo obtain better results, the prompt should be engineered. A basic recipe is the following:\n\n``raw prompt`` + ``style`` + ``artist`` + ``details``\n\n- Examples of ``style`` are: *Portrait*, *Realistic*, *Oil painting*, *Pencil drawing*, *Concept art*\n- Examples of ``artist`` are: Jan van Eyck (when ``style`` = *Portrait*), *Vincent Van Gogh* (when ``style`` = *Oil painting*), *Leonardo Da Vinci* (when ``style`` = *Pencil drawing*), and so on. Note that you can also mix artists, to get original results.\n- Examples of ``details`` are *Unreal Engine* if you want to add realistic lightining, *8 k* if you want to add more details, *artstation* if you want to make your image more artistic, and so on.\n\nExample of elaborated prompts: \n\n*\"Professional photograph of a lion with a black mane, high quality, highly detailed, award-winning, hd, 8k, awe-inspirin\"*\n\n\u003cdiv\u003e\n\u003cimg src=\"./img/lion_long1.png\" width=\"600\"/\u003e\n\u003c/div\u003e\n\n\n*\"retrofuturistic portrait of a lion in astro suit, space graphics art in background, close up, wlop, dan mumford, artgerm, liam brazier, peter mohrbacher, raw, featured in artstation, octane render, cinematic, elegant, intricate, 8 k\"*\n\n\u003cdiv\u003e\n\u003cimg src=\"./img/lion_long.png\" width=\"600\"/\u003e\n\u003c/div\u003e\n\nTo see more examples of prompts and get inspirations, check [here](https://lexica.art/). To find a prompt for a specific image, you can use [this](https://colab.research.google.com/github/pharmapsychotic/clip-interrogator/blob/main/clip_interrogator.ipynb?authuser=0\u0026pli=1#scrollTo=rbDEMDGJrJEo) image classifier notebook. \n\n\n## 3. 📚 Resources\n\n**Repositories**\n- A web interface with tons of advanced features that runs locally - [WebUI](https://github.com/AUTOMATIC1111/stable-diffusion-webui).\n- A WebUI extension to generate videos - [Deforum WebUI](https://github.com/deforum-art/sd-webui-deforum)\n\n\u003e [!WARNING] \n\u003e Since the libraries used in the notebooks below are often updated, the notebooks might stop working at some point. Please, open an issue if you encounter problems and bugs.\n\n**Colab notebooks (demo)**\n- text2img and img2img with advanced features [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1AfAmwLMd_Vx33O9IwY2TmO9wKZ8ABRRa)\n- Generate video animations [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/deforum/stable-diffusion/blob/main/Deforum_Stable_Diffusion.ipynb) (you need to download the weights from [here](https://huggingface.co/CompVis/stable-diffusion-v-1-4-original) and upload them to your Google Drive)\n- Find prompts with the interrogator [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pharmapsychotic/clip-interrogator/blob/main/clip_interrogator.ipynb?authuser=0\u0026pli=1#scrollTo=rbDEMDGJrJEo)\n- Stable Diffusion in Tensorflow/Keras [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1zVTa4mLeM_w44WaFwl7utTaa6JcaH1zK)\n- Image2Image pipeline for Stable Diffusion [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patil-suraj/Notebooks/blob/master/image_2_image_using_diffusers.ipynb)\n\n**Colab notebooks (tutorials)**\n- Introduction to diffusers 🧨, the Hugging Face 🤗 library for diffusion models [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/diffusers_intro.ipynb)\n- Introduction to Stable Diffusion with diffusers 🧨 [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/stable_diffusion.ipynb)\n- Training a diffusion model with diffusers 🧨 [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/gist/anton-l/f3a8206dae4125b93f05b1f5f703191d/diffusers_training_example.ipynb#scrollTo=smJeP67bF0yj)\n- Denoising Diffusion Implicit Models in Tensorflow/Keras [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/keras-team/keras-io/blob/master/examples/generative/ipynb/ddim.ipynb)\n\n\n**Blogs**\n- [What are Diffusion Models?](https://lilianweng.github.io/posts/2021-07-11-diffusion-models/#nice) introduction to Diffusion models and mathematical derivations.\n- [The Annotated Diffusion Model](https://huggingface.co/blog/annotated-diffusion) step-by-step tutorial for building a Diffusion model from scratch in PyTorch.\n- [Generative Modeling by Estimating Gradients of the Data Distribution](https://yang-song.net/blog/2021/score/) Introduction to score-based generative models.\n\n**Papers**\n- [[1](https://openaccess.thecvf.com/content/CVPR2022/papers/Rombach_High-Resolution_Image_Synthesis_With_Latent_Diffusion_Models_CVPR_2022_paper.pdf)] Rombach, Robin, et al. \"High-resolution image synthesis with latent diffusion models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.\n- [[2](https://arxiv.org/abs/2006.11239)] Ho, Jonathan, Ajay Jain, and Pieter Abbeel. \"Denoising diffusion probabilistic models.\" Advances in Neural Information Processing Systems, 2020.\n- [[3](https://arxiv.org/abs/1907.05600)] Song, Yang, and Stefano Ermon. \"Generative modeling by estimating gradients of the data distribution.\" Advances in Neural Information Processing Systems, 2019.\n","funding_links":[],"categories":["Tutorial and Jupyter Notebook"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FFilippoMB%2FDiffusion_models_tutorial","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FFilippoMB%2FDiffusion_models_tutorial","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FFilippoMB%2FDiffusion_models_tutorial/lists"}