https://github.com/neonwatty/yolo-diffusion

Demo of stable diffusion 2.0 applied to Yolo8 detected / segmented objects
https://github.com/neonwatty/yolo-diffusion

generative-art object-detection python pytorch stable-diffusion yolo8

Last synced: 22 days ago
JSON representation

Demo of stable diffusion 2.0 applied to Yolo8 detected / segmented objects

• Host: GitHub
• URL: https://github.com/neonwatty/yolo-diffusion
• Owner: neonwatty
• License: apache-2.0
• Created: 2023-06-11T14:06:42.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2023-06-20T17:00:37.000Z (over 1 year ago)
• Last Synced: 2024-10-10T19:15:37.558Z (about 1 month ago)
• Topics: generative-art, object-detection, python, pytorch, stable-diffusion, yolo8
• Language: Python
• Homepage:
• Size: 7.84 MB
• Stars: 6
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# yolo-diffusion

[![collab sticker](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/jermwatt/yolo_diffusion/blob/main/object_diffusion_demo.ipynb)

A simple demo for quick experimentation with [Stable Diffusion 2.0](https://huggingface.co/stabilityai/stable-diffusion-2-inpainting) applied to specific objects in 2d images, detected and segmented via [YOLOv8](https://docs.ultralytics.com/).

You can get started in [collab now](https://colab.research.google.com/github/jermwatt/yolo_diffusion/blob/main/object_diffusion_demo.ipynb) - or watch the walkthrough videos below first.

## Overview

With this simple demo you can start with any image - like the one on the left below.

You can then select any object in the image - like the `pizza`.  YOLO will detect and segment this object, and then you can replace it with a diffused suggestion via Stable v2.0.  

![cat cash](https://github.com/jermwatt/yolo-diffusion/blob/main/test_data/cat_cash.png?raw=true)

Like "a pile of cash" - shown on the right.

You can take the same image and do the same thing with another object - like the `cat` - as shown below.

![squirrel pizza](https://github.com/jermwatt/yolo-diffusion/blob/main/test_data/squirrel_pizza.png?raw=true)

Here we used the prompt "'sinister looking cartoon squirrel wearing sunglasses, high resolution'".

Here's another example.  In this case we replace the `person` with 'an ape, smiling, high resolution, holding something"

![squirrel pizza](https://github.com/jermwatt/yolo-diffusion/blob/main/test_data/example_person_replacement.png?raw=true)

YOLOv8 has more than 70 pre-trained objects that can be quickly detected / segmented, then fed to Stable Diffusion.  See the [demo notebook](https://colab.research.google.com/github/jermwatt/yolo-diffusion/blob/main/object_diffusion_demo.ipynb#scrollTo=8GhUUsG2tdZ1) for a complete list of available objects.