https://github.com/MichWozPol/LEGO_StableDiffusion

The project aim was to fine-tune the stable diffusion model in order to generate images in the LEGO style based on the prompt.
https://github.com/MichWozPol/LEGO_StableDiffusion

fine-tuning image-generation machine-learning stable-diffusion

Last synced: 3 months ago
JSON representation

The project aim was to fine-tune the stable diffusion model in order to generate images in the LEGO style based on the prompt.

• Host: GitHub
• URL: https://github.com/MichWozPol/LEGO_StableDiffusion
• Owner: MichWozPol
• Created: 2022-12-13T21:25:52.000Z (about 2 years ago)
• Default Branch: main
• Last Pushed: 2023-06-07T22:20:12.000Z (over 1 year ago)
• Last Synced: 2024-08-03T23:12:33.367Z (6 months ago)
• Topics: fine-tuning, image-generation, machine-learning, stable-diffusion
• Language: Python
• Homepage:
• Size: 9.28 MB
• Stars: 13
• Watchers: 1
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

• awesome-lego-machine-learning - LEGO Stable Diffusion [2023.01 - Fine-tuned stable diffusion model for generating images in the LEGO style. (Generating images of LEGO / Posts)

README

        
# LEGO_StableDiffusion

Overview

The project aim was to fine-tuned stable diffusion model in order to generate images in the LEGO style based on the prompt.


Fine-tuned model is created on top of Stable Diffusion text to image [model](https://huggingface.co/spaces/stabilityai/stable-diffusion).

To run app install requirements by typing: `pip3 install -r requirements`, then run the main app file `python3 app/main.py`, making sure you are in the root repo directory.

Database was created by our own by scraping the LEGO pictures from websites such as gettyimages.com, istockphoto.com and brickset.com (scripts available in the `Preprocessing` directory). LEGO database with mixed figures and constructions images is available  here. 


Image captioning was done using BLIP [model](https://github.com/salesforce/BLIP). Image captioning script (`image_captioning.py`) is available in the `Preprocessing/BLIP` project directory.


App is created using [gradio](https://gradio.app) library. In order to change GPU index or models' names do it in the `app/main.py` file.

Models are available here: https://www.kaggle.com/datasets/michasakowicz/lego-dataset-models.

Results

Images:


Rubber Duck Aliens visiting
 the Earth for the first time | A small cabin on top of a snowy
 mountain in the style of Disney | A high tech solarpunk utopia
 in the Amazon rainforest | An insect robot preparing 
a delicious meal

:-------------------------:|:-------------------------:|:-------------------------:|:-------------------------:

![alt-text-1](results/img1.jpg "Rubber Duck Aliens visiting the Earth for the first time") | ![alt-text-1](results/img2.jpg "A small cabin on top of a snowy mountain in the style of Disney") | ![alt-text-1](results/img3.jpg "A high tech solarpunk utopia in the Amazon rainforest") | ![alt-text-1](results/img4.jpg "An insect robot preparing a delicious meal​​")

App: