https://github.com/abhishekn1947/ghibli-diffuser-open-source
An open-source tool that transforms images into Studio Ghibli-style artwork using Stable Diffusion and LoRA. Pre-trained model and datasets included
https://github.com/abhishekn1947/ghibli-diffuser-open-source
ai-art-generator deep-learning fine image-transformation llm lora open-source python3 pytorch stable-diffusion studio-ghibli tuned
Last synced: about 1 month ago
JSON representation
An open-source tool that transforms images into Studio Ghibli-style artwork using Stable Diffusion and LoRA. Pre-trained model and datasets included
- Host: GitHub
- URL: https://github.com/abhishekn1947/ghibli-diffuser-open-source
- Owner: Abhishekn1947
- License: mit
- Created: 2025-03-30T02:49:46.000Z (2 months ago)
- Default Branch: main
- Last Pushed: 2025-04-27T09:23:07.000Z (about 1 month ago)
- Last Synced: 2025-05-04T08:58:40.257Z (about 1 month ago)
- Topics: ai-art-generator, deep-learning, fine, image-transformation, llm, lora, open-source, python3, pytorch, stable-diffusion, studio-ghibli, tuned
- Language: Jupyter Notebook
- Homepage:
- Size: 710 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# GhibliFilterTool
Welcome to **GhibliFilterTool**, an open-source project that harnesses Stable Diffusion and LoRA to bring the whimsical, hand-painted charm of Studio Ghibli to your images. Trained on an NVIDIA A100 GPU in Google Colab Pro, this tool is ready for you to explore, enhance, and collaborate on. I’m open-sourcing this to the GitHub community—fork it, improve it, and let’s take it further together. Cheers to the AI open-source community! 🎉
- I urge the open-source AI community to enhance and refine this model, ensuring AI remains accessible to everyone. To achieve this, we need greater GPU power, increased computing capacity, and more extensive training data.
## 🌟 Features
- **Ghibli Magic**: Turn any photo into a vibrant, pastel-colored masterpiece with lush backgrounds and whimsical details.
- **Flexible**: Tweak strength, guidance scale, and prompts for custom results.
- **Community-Driven**: Open for collaboration and improvement.
- **Colab-Friendly**: Easy to run in Google Colab with GPU support.
## 🎨 Examples
See GhibliFilterTool in action:
| Before | After |
|--------|-------|
| |  |
| |  |
| | 
## Architecture
## 🚀 Getting Started
## Drive Link to Datasets
-https://drive.google.com/drive/folders/1mxNdEK88t2OSdE5b30eMws4e7HoXRsGr?usp=sharing
-https://drive.google.com/drive/folders/1VI_N3IorCHcrtf_QwmYVdlI1LT3kCa4X?usp=sharing
## Drive Link to already fine tuned model for the above datasets:
-https://drive.google.com/file/d/13wbRL1hFrjUREBR2J-qW-HUWvecEDdzY/view?usp=sharing
### Running in Google Colab with A100 GPU
Run GhibliFilterTool in Google Colab with a GPU (A100 recommended, but T4/V100 work too). Here’s how:
1. Open Colab:
- Head to [Google Colab](https://colab.research.google.com/) and start a new notebook.
2. Set GPU Runtime:
- Go to `Runtime` > `Change runtime type` > Select `GPU A100` > Save.
3. Clone the Repo: git clone https://github.com/[YourUsername]/GhibliFilterTool.git
**Install Dependencies**: !pip install torch torchvision diffusers peft python-dotenv pillow
**Upload Datasets**: Upload ghibli_dataset (1022 Ghibli images) and regularization_dataset (1000 + generic photos) via Colab’s file uploader or a cloud drive link. (i've included the folders containing the images as required)
**Run the Code**: Copy the notebook cells from GhibliFilterTool.ipynb (or use snippets below) and execute them step-by-step.
**Generate Art**: Upload an image in the final cell to see the Ghibli transformation!
## 🤝 Contributing
I’m open to collaboration and excited to see where the community takes this! Here’s how you can jump in:
## Fork & Enhance
**Fork the repo, tweak the model (e.g., more epochs, LoRA settings), and submit a pull request.**
## Improve the Model
**Experiment with dataset size, hyperparameters, or inference options to perfect the Ghibli style.**
## Share Ideas
-Open an issue for feature suggestions, bug reports, or discussions.
**I trained this on an A100 GPU in Colab Pro, but you’re welcome to adapt it to other setups. Let’s make this a collaborative masterpiece—cheers to open-source AI! 🌍**
## 📖 Technical Details Overview:
## Current Model
**GhibliFilterTool fine-tunes Stable Diffusion v1.5 with LoRA to apply Studio Ghibli’s aesthetic. It uses 1022 Ghibli images for style and 250 regularization images for balance.**
## Training Setup:
Hardware: NVIDIA A100 GPU (Colab Pro)
Dataset: 1022 Ghibli images, 250 regularization images
Epochs: 15 (early stopping at 6)
Batch Size: 4 (effective 8 with gradient accumulation)
LoRA: r=48, lora_alpha=96, target_modules=["to_q", "to_k", "to_v"]
LR: 5e-5, step decay every 7 epochs
Instance Weight: 2.0
## Latest Training Log:
Starting fine-tuning with 1022 instance images and 250 regularization images...
Epoch 1/15 completed. Avg Instance Loss: 0.1402, Avg Reg Loss: 0.1572, LR: 0.000030
Epoch 2/15 completed. Avg Instance Loss: 0.1446, Avg Reg Loss: 0.1520, LR: 0.000030
Epoch 3/15 completed. Avg Instance Loss: 0.1446, Avg Reg Loss: 0.1507, LR: 0.000030
Epoch 4/15 completed. Avg Instance Loss: 0.1405, Avg Reg Loss: 0.1583, LR: 0.000030
Epoch 5/15 completed. Avg Instance Loss: 0.1462, Avg Reg Loss: 0.1515, LR: 0.000015
Epoch 6/15 completed. Avg Instance Loss: 0.1484, Avg Reg Loss: 0.1569, LR: 0.000015
Early stopping triggered after 6 epochs.
Performance: Instance loss ~0.14 (target <0.10). Output is decent but needs stronger Ghibli style.
## Improvement Ideas:
-Boost LoRA: Try r=64, lora_alpha=128 on high-VRAM GPUs.
-More Epochs: Extend to 20–25 with patience=10.
-Dataset: Add more Ghibli images (2000+) or curate for quality.
-Tuning: Test lr=1e-4, instance_weight=3.0, or larger effective batch sizes.
-Inference: Experiment with strength=0.8–1.0, guidance_scale=9.0–10.0.
## 📜 License
**MIT License—use, modify, and share freely!**
## 🙌 Acknowledgements
**AI Community**: For tools and inspiration.
**Hugging Face**: For Stable Diffusion and diffusers.
**Studio Ghibli**: For the art that sparked this journey and my personal interest in Anime!
**Let’s collaborate and bring more Ghibli magic to life! ✨**