{"id":13646061,"url":"https://github.com/huggingface/instruction-tuned-sd","last_synced_at":"2025-10-14T15:32:40.904Z","repository":{"id":167676439,"uuid":"615140875","full_name":"huggingface/instruction-tuned-sd","owner":"huggingface","description":"Code for instruction-tuning Stable Diffusion.","archived":false,"fork":false,"pushed_at":"2024-02-16T09:09:24.000Z","size":108,"stargazers_count":241,"open_issues_count":5,"forks_count":19,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-09-30T18:02:28.775Z","etag":null,"topics":["accelerate","diffusers","generative-ai","image-editing","instruction-tuning","stable-diffusion","transformers"],"latest_commit_sha":null,"homepage":"https://huggingface.co/blog/instruction-tuning-sd","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/huggingface.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-03-17T03:13:25.000Z","updated_at":"2025-09-23T03:15:37.000Z","dependencies_parsed_at":"2023-11-26T10:26:49.918Z","dependency_job_id":"6ec5f4dc-4f06-408c-9f68-604771771abb","html_url":"https://github.com/huggingface/instruction-tuned-sd","commit_stats":null,"previous_names":["huggingface/instruction-tuned-sd"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/huggingface/instruction-tuned-sd","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Finstruction-tuned-sd","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Finstruction-tuned-sd/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Finstruction-tuned-sd/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Finstruction-tuned-sd/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/huggingface","download_url":"https://codeload.github.com/huggingface/instruction-tuned-sd/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Finstruction-tuned-sd/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279019322,"owners_count":26086711,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-14T02:00:06.444Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["accelerate","diffusers","generative-ai","image-editing","instruction-tuning","stable-diffusion","transformers"],"created_at":"2024-08-02T01:02:47.690Z","updated_at":"2025-10-14T15:32:40.889Z","avatar_url":"https://github.com/huggingface.png","language":"Python","funding_links":[],"categories":["Others"],"sub_categories":[],"readme":"# Instruction-tuning Stable Diffusion\n\n**TL;DR**: Motivated partly by [FLAN](https://arxiv.org/abs/2109.01652) and partly by [InstructPix2Pix](https://arxiv.org/abs/2211.09800), we explore a way to instruction-tune [Stable Diffusion](https://stability.ai/blog/stable-diffusion-public-release). This allows us to prompt our model using an input image and an “instruction”, such as - *Apply a cartoon filter to the natural image*.\n\nYou can read [our blog post](https://hf.co/blog/instruction-tuning-sd) to know more details. \n\n## Table of contents\n\n🐶 [Motivation](#motivation) \u003cbr\u003e\n📷 [Data preparation](#data-preparation) \u003cbr\u003e\n💺 [Training](#training) \u003cbr\u003e\n🎛 [Models, datasets, demo](#models-datasets-demo) \u003cbr\u003e\n⭐️ [Inference](#inference) \u003cbr\u003e\n🧭 [Results](#results) \u003cbr\u003e\n🤝 [Acknowledgements](#acknowledgements) \u003cbr\u003e\n\n## Motivation \n\nInstruction-tuning is a supervised way of teaching language models to follow instructions to solve a task. It was introduced in [Fine-tuned Language Models Are Zero-Shot Learners](https://arxiv.org/abs/2109.01652) (FLAN) by Google. From recent times, you might recall works like [Alpaca](https://crfm.stanford.edu/2023/03/13/alpaca.html) and [FLAN V2](https://arxiv.org/abs/2210.11416), which are good examples of how beneficial instruction-tuning can be for various tasks. \n\nOn the other hand, the idea of teaching Stable Diffusion to follow user instructions to perform edits on input images was introduced in [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://arxiv.org/abs/2211.09800). \n\nOur motivation behind this work comes partly from the FLAN line of works and partly from InstructPix2Pix. We wanted to explore if it’s possible to prompt Stable Diffusion with specific instructions and input images to process them as per our needs. \n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/instruction-tuning-sd.png\" width=600/\u003e\n\u003c/p\u003e\n\nOur main idea is to first create an instruction prompted dataset (as described in [our blog](https://hf.co/blog/instruction-tuning-sd) and then conduct InstructPix2Pix style training. The end objective is to make Stable Diffusion better at following specific instructions that entail image transformation related operations.\n\n\n## Data preparation\n\nOur data preparation process is inspired by FLAN. Refer to the sections below for more details.\n\n* **Cartoonization**: Refer to the `data_preparation` directory.\n* **Low-level image processing**: Refer to the [dataset card](https://huggingface.co/datasets/instruction-tuning-sd/low-level-image-proc).\n\n## Training\n\n\u003e [!TIP]\n\u003e In case of using custom datasets, one needs to configure the dataset as per their choice as long as you maintain the format presented here. You might have to configure your dataloader and dataset class in case you don't want to make use of the `datasets` library. If you do so, you might have to adjust the training scripts accordingly.\n\n### Dev env setup\n\nWe recommend using a Python virtual environment for this. Feel free to use your favorite one here. \n\nWe conducted our experiments with PyTorch 1.13.1 (CUDA 11.6) and a single A100 GPU. Since PyTorch installation can be hardware-dependent, we refer you to the [official docs](https://pytorch.org/) for installing PyTorch. \n\nOnce PyTorch is installed, we can install the rest of the dependencies:\n\n```bash \npip install -r requirements.txt\n```\n\nAdditionally, we recommend installing [xformers](https://github.com/facebookresearch/xformers) as well for enabling memory-efficient training.\n\n\u003e 💡 **Note**: If you're using PyTorch 2.0 then you don't need to additionally install xformers. This is because we default to a memory-efficient attention processor in Diffusers when PyTorch 2.0 is being used. \n\n### Launching training\n\nOur training code leverages [🧨 diffusers](https://github.com/huggingface/diffusers), [🤗 accelerate](https://github.com/huggingface/accelerate), and [🤗 transformers](https://github.com/huggingface/transformers). In particular, we extend [this training example](https://github.com/huggingface/diffusers/blob/main/examples/instruct_pix2pix/train_instruct_pix2pix.py) to fit our needs. \n\n### Cartoonization\n\n#### Training from scratch using the InstructPix2Pix methodology\n\n```bash \nexport MODEL_ID=\"runwayml/stable-diffusion-v1-5\"\nexport DATASET_ID=\"instruction-tuning-sd/cartoonization\"\nexport OUTPUT_DIR=\"cartoonization-scratch\"\n\naccelerate launch --mixed_precision=\"fp16\" train_instruct_pix2pix.py \\\n  --pretrained_model_name_or_path=$MODEL_ID \\\n  --dataset_name=$DATASET_ID \\\n  --use_ema \\\n  --enable_xformers_memory_efficient_attention \\\n  --resolution=256 --random_flip \\\n  --train_batch_size=2 --gradient_accumulation_steps=4 --gradient_checkpointing \\\n  --max_train_steps=15000 \\\n  --checkpointing_steps=5000 --checkpoints_total_limit=1 \\\n  --learning_rate=5e-05 --lr_warmup_steps=0 \\\n  --mixed_precision=fp16 \\\n  --val_image_url=\"https://hf.co/datasets/diffusers/diffusers-images-docs/resolve/main/mountain.png\" \\\n  --validation_prompt=\"Generate a cartoonized version of the natural image\" \\\n  --seed=42 \\\n  --output_dir=$OUTPUT_DIR \\\n  --report_to=wandb \\\n  --push_to_hub\n```\n\n\u003e 💡 **Note**: Following InstructPix2Pix, we train on the 256x256 resolution and that doesn't seem to affect the end quality too much when we perform inference with the 512x512 resolution.\n\nOnce the training successfully launched, the logs will be automatically tracked using Weights and Biases. Depending on how you specified the `checkpointing_steps` and the `max_train_steps`, there will be intermediate checkpoints too. At the end of training, you can expect a directory (namely `OUTPUT_DIR`) that contains the intermediate checkpoints and the final pipeline artifacts. \n\nIf `--push_to_hub` is specified, the contents of `OUTPUT_DIR` will be pushed to a repository on the Hugging Face Hub. \n\n[Here](https://wandb.ai/sayakpaul/instruction-tuning-sd/runs/wszjpb1b) is an example run page on Weights and Biases. [Here](https://huggingface.co/instruction-tuning-sd/scratch-cartoonizer) is an example of how the pipeline repository would look like on the Hugging Face Hub. \n\n#### Fine-tuning from InstructPix2Pix\n\n```bash \nexport MODEL_ID=\"timbrooks/instruct-pix2pix\"\nexport DATASET_ID=\"instruction-tuning-sd/cartoonization\"\nexport OUTPUT_DIR=\"cartoonization-finetuned\"\n\naccelerate launch --mixed_precision=\"fp16\" finetune_instruct_pix2pix.py \\\n  --pretrained_model_name_or_path=$MODEL_ID \\\n  --dataset_name=$DATASET_ID \\\n  --use_ema \\\n  --enable_xformers_memory_efficient_attention \\\n  --resolution=256 --random_flip \\\n  --train_batch_size=2 --gradient_accumulation_steps=4 --gradient_checkpointing \\\n  --max_train_steps=15000 \\\n  --checkpointing_steps=5000 --checkpoints_total_limit=1 \\\n  --learning_rate=5e-05 --lr_warmup_steps=0 \\\n  --mixed_precision=fp16 \\\n  --val_image_url=\"https://hf.co/datasets/diffusers/diffusers-images-docs/resolve/main/mountain.png\" \\\n  --validation_prompt=\"Generate a cartoonized version of the natural image\" \\\n  --seed=42 \\\n  --output_dir=$OUTPUT_DIR \\\n  --report_to=wandb \\\n  --push_to_hub\n```\n\n### Low-level image processing\n\n#### Training from scratch using the InstructPix2Pix methodology\n\n```bash \nexport MODEL_ID=\"runwayml/stable-diffusion-v1-5\"\nexport DATASET_ID=\"instruction-tuning-sd/low-level-image-proc\"\nexport OUTPUT_DIR=\"low-level-img-proc-scratch\"\n\naccelerate launch --mixed_precision=\"fp16\" train_instruct_pix2pix.py \\\n  --pretrained_model_name_or_path=$MODEL_ID \\\n  --dataset_name=$DATASET_ID \\\n  --original_image_column=\"input_image\" \\\n  --edit_prompt_column=\"instruction\" \\\n  --edited_image_column=\"ground_truth_image\" \\\n  --use_ema \\\n  --enable_xformers_memory_efficient_attention \\\n  --resolution=256 --random_flip \\\n  --train_batch_size=2 --gradient_accumulation_steps=4 --gradient_checkpointing \\\n  --max_train_steps=15000 \\\n  --checkpointing_steps=5000 --checkpoints_total_limit=1 \\\n  --learning_rate=5e-05 --lr_warmup_steps=0 \\\n  --mixed_precision=fp16 \\\n  --val_image_url=\"https://hf.co/datasets/sayakpaul/sample-datasets/resolve/main/derain_the_image_1.png\" \\\n  --validation_prompt=\"Derain the image\" \\\n  --seed=42 \\\n  --output_dir=$OUTPUT_DIR \\\n  --report_to=wandb \\\n  --push_to_hub\n```\n\n#### Fine-tuning from InstructPix2Pix\n\n```bash \nexport MODEL_ID=\"timbrooks/instruct-pix2pix\"\nexport DATASET_ID=\"instruction-tuning-sd/low-level-image-proc\"\nexport OUTPUT_DIR=\"low-level-img-proc-finetuned\"\n\naccelerate launch --mixed_precision=\"fp16\" finetune_instruct_pix2pix.py \\\n  --pretrained_model_name_or_path=$MODEL_ID \\\n  --dataset_name=$DATASET_ID \\\n  --original_image_column=\"input_image\" \\\n  --edit_prompt_column=\"instruction\" \\\n  --edited_image_column=\"ground_truth_image\" \\\n  --use_ema \\\n  --enable_xformers_memory_efficient_attention \\\n  --resolution=256 --random_flip \\\n  --train_batch_size=2 --gradient_accumulation_steps=4 --gradient_checkpointing \\\n  --max_train_steps=15000 \\\n  --checkpointing_steps=5000 --checkpoints_total_limit=1 \\\n  --learning_rate=5e-05 --lr_warmup_steps=0 \\\n  --mixed_precision=fp16 \\\n  --val_image_url=\"https://hf.co/datasets/sayakpaul/sample-datasets/resolve/main/derain_the_image_1.png\" \\\n  --validation_prompt=\"Derain the image\" \\\n  --seed=42 \\\n  --output_dir=$OUTPUT_DIR \\\n  --report_to=wandb \\\n  --push_to_hub\n```\n\n## Models, datasets, demo\n\n### **Models**:\n  * [instruction-tuning-sd/scratch-low-level-img-proc](https://huggingface.co/instruction-tuning-sd/scratch-low-level-img-proc)\n  * [instruction-tuning-sd/scratch-cartoonizer](https://huggingface.co/instruction-tuning-sd/scratch-cartoonizer)\n  * [instruction-tuning-sd/cartoonizer](https://huggingface.co/instruction-tuning-sd/cartoonizer)\n  * [instruction-tuning-sd/low-level-img-proc](https://huggingface.co/instruction-tuning-sd/low-level-img-proc)\n\n### **Datasets**:\n  * [Instruction-prompted cartoonization](https://huggingface.co/datasets/instruction-tuning-sd/cartoonization)\n  * [Instruction-prompted low-level image processing](https://huggingface.co/datasets/instruction-tuning-sd/low-level-image-proc) \n\n### Demo on 🤗 Spaces\n\nTry out the models interactively WITHOUT any setup: [Demo](https://huggingface.co/spaces/instruction-tuning-sd/instruction-tuned-sd)\n\n## Inference\n\n### Cartoonization\n\n```python\nimport torch\nfrom diffusers import StableDiffusionInstructPix2PixPipeline\nfrom diffusers.utils import load_image\n\nmodel_id = \"instruction-tuning-sd/cartoonizer\"\npipeline = StableDiffusionInstructPix2PixPipeline.from_pretrained(\n    model_id, torch_dtype=torch.float16, use_auth_token=True\n).to(\"cuda\")\n\nimage_path = \"https://hf.co/datasets/diffusers/diffusers-images-docs/resolve/main/mountain.png\"\nimage = load_image(image_path)\n\nimage = pipeline(\"Cartoonize the following image\", image=image).images[0]\nimage.save(\"image.png\")\n```\n\n### Low-level image processing \n\n```python \nimport torch\nfrom diffusers import StableDiffusionInstructPix2PixPipeline\nfrom diffusers.utils import load_image\n\nmodel_id = \"instruction-tuning-sd/low-level-img-proc\"\npipeline = StableDiffusionInstructPix2PixPipeline.from_pretrained(\n    model_id, torch_dtype=torch.float16, use_auth_token=True\n).to(\"cuda\")\n\nimage_path = \"https://hf.co/datasets/sayakpaul/sample-datasets/resolve/main/derain%20the%20image_1.png\"\nimage = load_image(image_path)\n\nimage = pipeline(\"derain the image\", image=image).images[0]\nimage.save(\"image.png\")\n```\n\n\n\u003e 💡 **Note**: Since the above pipelines are essentially of type `StableDiffusionInstructPix2PixPipeline`, you can customize several arguments that\nthe pipeline exposes. Refer to the [official docs](https://huggingface.co/docs/diffusers/main/en/api/pipelines/stable_diffusion/pix2pix) for more details.\n\n## Results\n\n### Cartoonization\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://i.imgur.com/wOCjpdI.jpg\"/\u003e\n\u003c/p\u003e\n\n---\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://i.imgur.com/RhTG8Lf.jpg\"/\u003e\n\u003c/p\u003e\n\n### Low-level image processing\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://i.imgur.com/LOhcJLv.jpg\"/\u003e\n\u003c/p\u003e\n\n---\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://i.imgur.com/uhTqIpY.png\"/\u003e\n\u003c/p\u003e\n\nRefer to our [blog post](https://hf.co/blog/instruction-tuning-sd) for more discussions on results and open questions.  \n\n\n## Acknowledgements\n\nThanks to [Alara Dirik](https://www.linkedin.com/in/alaradirik/) and [Zhengzhong Tu](https://www.linkedin.com/in/zhengzhongtu) for the helpful discussions. \n\n## Citation\n\n```bibtex\n@article{\n  Paul2023instruction-tuning-sd,\n  author = {Paul, Sayak},\n  title = {Instruction-tuning Stable Diffusion with InstructPix2Pix},\n  journal = {Hugging Face Blog},\n  year = {2023},\n  note = {https://huggingface.co/blog/instruction-tuning-sd},\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuggingface%2Finstruction-tuned-sd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhuggingface%2Finstruction-tuned-sd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuggingface%2Finstruction-tuned-sd/lists"}