{"id":13778815,"url":"https://github.com/mikonvergence/DiffusionFastForward","last_synced_at":"2025-05-11T12:32:00.204Z","repository":{"id":142489313,"uuid":"597490428","full_name":"mikonvergence/DiffusionFastForward","owner":"mikonvergence","description":"DiffusionFastForward: a free course and experimental framework for diffusion-based generative models","archived":false,"fork":false,"pushed_at":"2023-06-11T07:31:40.000Z","size":4287,"stargazers_count":552,"open_issues_count":3,"forks_count":53,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-08-03T18:13:24.345Z","etag":null,"topics":["diffusion-model","diffusion-models","generative-art","generative-model","generative-models","image-generation","latent-diffusion","learning-resources"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mikonvergence.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-02-04T18:00:32.000Z","updated_at":"2024-08-03T13:38:57.000Z","dependencies_parsed_at":"2023-06-26T02:08:07.278Z","dependency_job_id":null,"html_url":"https://github.com/mikonvergence/DiffusionFastForward","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikonvergence%2FDiffusionFastForward","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikonvergence%2FDiffusionFastForward/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikonvergence%2FDiffusionFastForward/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikonvergence%2FDiffusionFastForward/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mikonvergence","download_url":"https://codeload.github.com/mikonvergence/DiffusionFastForward/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225048994,"owners_count":17412906,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["diffusion-model","diffusion-models","generative-art","generative-model","generative-models","image-generation","latent-diffusion","learning-resources"],"created_at":"2024-08-03T18:00:57.774Z","updated_at":"2024-11-17T14:30:58.431Z","avatar_url":"https://github.com/mikonvergence.png","language":"Jupyter Notebook","readme":"![Asset 5](https://user-images.githubusercontent.com/13435425/222425743-213279f9-d0a1-413c-a16a-2c88b512f827.png)\n\n### :rocket: Diffusion models are making the headlines as a new generation of powerful generative models.\n\nHowever, many of the ongoing research considers solutions that are quite often **quite specific** and **require large computational resources for training**.\n\n:beginner: **DiffusionFastForward** offers a general template for diffusion models for images that can be a starting point for understanding and researching diffusion-based generative models.\n* :zap: **PyTorch Lightning** to enable easy training!\n* :money_with_wings: You can run all experiments online on Google colab - **no need for own GPU machine**!\n* :mag_right: Examples for **both low-resolution and high-resolution** data!\n* :tent: Examples of **latent diffusion**!\n* :art: Examples of **image translation** with diffusion!\n\nThe code structure is simple, so that you can easily customize it to your own applications.\n\n\u003e :construction: Disclaimer: This repository does not provide any weights to the models. The purpose of this software is to be able to train **new** weights on a previously unexplored type of data.\n\n## Contents\n\nThere are three elements integrated into this project:\n* :computer: **Code**\n* :bulb: **Notes** (in `notes` directory)\n* :tv: **Video Course** (to be released on YouTube)\n\n---\n### :computer: [**Code**](https://github.com/mikonvergence/DiffusionFastForward/tree/master/src)\nThis repository offers a starting point for training diffusion models on new types of data. It can serve as a baseline that can hopefully be developed into more robust solutions based on the specific features of the performed generative task.\n\nIt includes notebooks that can be run stand-alone:\n1. [![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/mikonvergence/DiffusionFastForward/blob/master/01-Diffusion-Sandbox-colab.ipynb) [01-Diffusion-Sandbox](https://github.com/mikonvergence/DiffusionFastForward/blob/master/01-Diffusion-Sandbox.ipynb)  - visualizations of the diffusion process \n2. [![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/mikonvergence/DiffusionFastForward/blob/master/02-Pixel-Diffusion-colab.ipynb) [02-Pixel-Diffusion](https://github.com/mikonvergence/DiffusionFastForward/blob/master/02-Pixel-Diffusion.ipynb) - basic diffusion suitable for low-resolution data\n3. [![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/mikonvergence/DiffusionFastForward/blob/master/03-Conditional-Pixel-Diffusion-colab.ipynb) [03-Conditional-Pixel-Diffusion](https://github.com/mikonvergence/DiffusionFastForward/blob/master/03-Conditional-Pixel-Diffusion.ipynb) - image translation with diffusion for low-resolution data\n4. [![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/mikonvergence/DiffusionFastForward/blob/master/04-Latent-Diffusion-colab.ipynb) [04-Latent-Diffusion](https://github.com/mikonvergence/DiffusionFastForward/blob/master/04-Latent-Diffusion.ipynb) - latent diffusion suitable for high-resolution data\n5. [![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/mikonvergence/DiffusionFastForward/blob/master/05-Conditional-Latent-Diffusion-colab.ipynb) [05-Conditional-Latent-Diffusion](https://github.com/mikonvergence/DiffusionFastForward/blob/master/05-Conditional-Latent-Diffusion.ipynb) - image translation with latent diffusion\n\n### Dependencies\nAssuming `torch` and `torchvision` is installed:\n```bash\npip install pytorch-lightning==1.9.3 diffusers einops\n```\n\n---\n### :bulb: [**Notes**](https://github.com/mikonvergence/DiffusionFastForward/tree/master/notes)\nShort summary notes are released as part of this repository and they overlap semantically with the notebooks!\n1. [01-Diffusion-Theory](https://github.com/mikonvergence/DiffusionFastForward/blob/master/notes/01-Diffusion-Theory.md) - visualizations of the diffusion process\n2. [02-Pixel-Diffusion](https://github.com/mikonvergence/DiffusionFastForward/blob/master/notes/02-Pixel-Diffusion.md) - basic diffusion suitable for low-resolution data\n3. [03-Conditional-Pixel-Diffusion](https://github.com/mikonvergence/DiffusionFastForward/blob/master/notes/03-Conditional-Pixel-Diffusion.md) - image translation with diffusion for low-resolution data\n4. [04-Latent-Diffusion](https://github.com/mikonvergence/DiffusionFastForward/blob/master/notes/04-Latent-Diffusion.md) - latent diffusion suitable for high-resolution data\n5. [05-Conditional-Latent-Diffusion](https://github.com/mikonvergence/DiffusionFastForward/blob/master/notes/05-Conditional-Latent-Diffusion.md) - image translation with latent diffusion\n\n---\n### :tv: [**Video Course**](https://youtube.com/playlist?list=PL5RHjmn-MVHDMcqx-SI53mB7sFOqPK6gN) (released on YouTube)\nThe course is released on YouTube and provides an extension to this repository. Some additional topics are covered, such as seminal papers and on-going research work.\n\n[\u003cimg width=\"1596\" alt=\"Screenshot 2023-03-01 at 19 46 20\" src=\"https://user-images.githubusercontent.com/13435425/222248673-bfcce06c-0f5b-421b-92b2-b4ed130c0dfb.png\"\u003e](https://youtube.com/playlist?list=PL5RHjmn-MVHDMcqx-SI53mB7sFOqPK6gN)\n\nThe current plan for the video course (links added upon publishing):\n* :tv: [#00 Introduction](https://youtu.be/sXBPyKycOX8?list=PL5RHjmn-MVHDMcqx-SI53mB7sFOqPK6gN)\n* :tv: [#01 Basics: Denoising Diffusion Process](https://youtu.be/7d73Y30viC8?list=PL5RHjmn-MVHDMcqx-SI53mB7sFOqPK6gN)\n* :tv: [#02 Basics: Denoising Diffusion of Images](https://youtu.be/Whykds7oWQI?list=PL5RHjmn-MVHDMcqx-SI53mB7sFOqPK6gN)\n* :tv: [#03 Practical: Unconditional Diffusion in Low-Resolution](https://youtu.be/eVYrHiwqBbE?list=PL5RHjmn-MVHDMcqx-SI53mB7sFOqPK6gN)\n* :soon: [#04 Extra: Summary of Seminal Works](/)\n* :tv: [#05 Basics: Conditional Diffusion](https://youtu.be/KuKeM_NbdSM?list=PL5RHjmn-MVHDMcqx-SI53mB7sFOqPK6gN)\n* :tv: [#06 Practical: Condition Diffusion in Low-Resolution](https://youtu.be/C-HjC-Kh4M0?list=PL5RHjmn-MVHDMcqx-SI53mB7sFOqPK6gN)\n* :tv: [#07 Basics: High-Resolution Diffusion](https://youtu.be/QaasB5PYMYs?list=PL5RHjmn-MVHDMcqx-SI53mB7sFOqPK6gN)\n* :tv: [#08 Practical: High-Resolution Diffusion](https://youtu.be/SPOIBtYEIxw?list=PL5RHjmn-MVHDMcqx-SI53mB7sFOqPK6gN)\n* :soon: [#09 Extra: Diffusion Applications](/)\n* :soon: [#10 Extra: Further Insight into Diffusion](/)\n\n---\n\n### :moneybag: Training Cost\n\nMost examples are one of two types of models, trainable within a day:\n\n**PixelDiffusion (Good for small images :baby:)**\nAppropriate for LR data. Direct diffusion in pixel space.\n| **Image Resolution** | 64x64 |\n|---|---|\n| **Training Time** | ~10 hrs |\n| **Memory Usage**  | ~4 GB |\n\n![out-pixel-conditional-1](https://user-images.githubusercontent.com/13435425/216780421-3613e80f-312b-4fed-8e24-9be22902a40f.png)\n\n![out-pixel-conditional-2](https://user-images.githubusercontent.com/13435425/216780427-5f737606-6b34-4477-88d4-187b863df03a.png)\n\n![out-pixel-conditional-3](https://user-images.githubusercontent.com/13435425/216780431-e3cf6a78-edc4-4ec6-9d3f-8a38aa01d5e0.png)\n\n**LatentDiffusion (Good for large images :whale2:)**\nUseful for HR data. Latent diffusion in compressed space.\n| **Image Resolution** | 256x256 |\n|---|---|\n| **Training Time** | ~20 hrs |\n| **Memory Usage**  | ~5 GB |\n\n![out-latent-conditional-1](https://user-images.githubusercontent.com/13435425/216780511-1f2d60ec-2d7e-4b5c-b851-c330687e55c3.png)\n\n![out-latent-conditional-2](https://user-images.githubusercontent.com/13435425/216780522-cc20ad37-4bf9-4a1d-af59-75c697029553.png)\n\n![out-latent-conditional-3](https://user-images.githubusercontent.com/13435425/216780528-e155b541-4dab-453d-9958-72e6350ae659.png)\n\n---\n\n### Other Software Resources\nThere are many great projects focused on diffusion generative models. However, most of them involve somewhat complex frameworks that are not always suitable for learning and preliminary experimentation.\n\n* 🤗 [diffusers](https://github.com/huggingface/diffusers)\n* lucidrains [PyTorch DDPM](https://github.com/lucidrains/denoising-diffusion-pytorch)\n* OpenAI [guided-diffusion](https://github.com/openai/guided-diffusion)\n* OpenAI [improved-diffusion](https://github.com/openai/improved-diffusion)\n* CompVis [latent-diffusion](https://github.com/CompVis/latent-diffusion)\n* Meta [DiT](https://github.com/facebookresearch/DiT)\n* MONAI [GenerativeModels for Medical Imaging](https://github.com/Project-MONAI/GenerativeModels)\n\n### Other Educational Resources\nSome excellent materials have already been published on the topic! Huge respect to all of the creators :pray: - check them out if their work has helped you!\n\n:coffee: **Blog Posts**\n* [Score-based Perspective](https://yang-song.net/blog/2021/score/) by [Yang Song](https://yang-song.net)\n* [What are Diffusion Models?](https://lilianweng.github.io/posts/2021-07-11-diffusion-models/) by [Lilian Weng](https://twitter.com/lilianweng/)\n* [Annotated Diffusion Model](https://huggingface.co/blog/annotated-diffusion) by [Niels Rogge](https://twitter.com/NielsRogge) and [Kashif Rasul](https://twitter.com/krasul)\n* [Diffusion as a kind of VAE](https://angusturner.github.io/generative_models/2021/06/29/diffusion-probabilistic-models-I.html) by [Angus Turner](https://angusturner.github.io/about/)\n\n:crystal_ball: **Explanation Videos**\n* [Diffusion Model Math Explained](https://www.youtube.com/watch?v=HoKDTa5jHvg) by [Outlier](https://twitter.com/dome39931447)\n* [What are Diffusion Models?](https://www.youtube.com/watch?v=fbLgFrlTnGU) by [Ari Seff](https://www.ariseff.com)\n* [Diffusion Models Beat GANs on Image Synthesis](https://www.youtube.com/watch?v=W-O7AZNzbzQ) the research paper explained by [Yannic Kilcher](https://twitter.com/ykilcher)\n\n:wrench: **Implementation Videos**\n* [Diffusion Models PyTorch Implementation](https://www.youtube.com/watch?v=TBCRlnwJtZU) by [Outlier](https://twitter.com/dome39931447)\n* [High-Resolution Image Synthesis with LDMs | ML Coding Series](https://www.youtube.com/watch?v=f6PtJKdey8E) by [Aleksa Gordić](https://www.youtube.com/@TheAIEpiphany)\n\n:mortar_board: **Video Lectures/Tutorials**\n* [Diffusion Probabilistic Models](https://www.youtube.com/watch?v=XCUlnHP1TNM) - MIT 6.S192 lecture by [Jascha Sohl-Dickstein](http://www.sohldickstein.com)\n* [Generative art using diffusion](https://www.youtube.com/watch?v=xYJEvihz3OI) - MIT 6.S192 lecture by [Prafulla Dhariwal](https://prafulladhariwal.com)\n* [Learning to Generate Data by Estimating Gradients of the Data Distribution](https://www.youtube.com/watch?v=nv-WTeKRLl0) by [Yang Song](https://yang-song.net)\n* [Denoising Diffusion-based Generative Modeling: Foundations and Applications](https://www.youtube.com/watch?v=cS6JQpEY9cs) tutorial presented at CVPR2022 by [Karsten Kreis](https://twitter.com/karsten_kreis), [Ruiqi Gao](https://twitter.com/RuiqiGao) and [Arash Vahdat](https://twitter.com/ArashVahdat)\n* [Generative Modeling by Estimating Gradients of the Data Distribution](https://www.youtube.com/watch?v=8TcNXi3A5DI) by [Stefano Ermon](https://mobile.twitter.com/stefanoermon)\n* [Variational autoencoders and Diffusion Models](https://www.youtube.com/watch?v=pea3sH6orMc) by [Tim Salimans](https://mobile.twitter.com/TimSalimans)\n","funding_links":[],"categories":["HarmonyOS","Tutorial and Jupyter Notebook"],"sub_categories":["Windows Manager"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmikonvergence%2FDiffusionFastForward","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmikonvergence%2FDiffusionFastForward","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmikonvergence%2FDiffusionFastForward/lists"}