{"id":13411466,"url":"https://github.com/XavierXiao/Dreambooth-Stable-Diffusion","last_synced_at":"2025-03-14T17:30:53.033Z","repository":{"id":58711700,"uuid":"533199740","full_name":"XavierXiao/Dreambooth-Stable-Diffusion","owner":"XavierXiao","description":"Implementation of Dreambooth (https://arxiv.org/abs/2208.12242) with Stable Diffusion","archived":false,"fork":false,"pushed_at":"2022-12-08T02:19:03.000Z","size":5985,"stargazers_count":7595,"open_issues_count":137,"forks_count":795,"subscribers_count":92,"default_branch":"main","last_synced_at":"2024-10-29T15:33:25.448Z","etag":null,"topics":["pytorch","pytorch-lightning","stable-diffusion","text-to-image"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/XavierXiao.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-09-06T06:57:40.000Z","updated_at":"2024-10-29T07:05:36.000Z","dependencies_parsed_at":"2022-09-06T16:11:52.054Z","dependency_job_id":null,"html_url":"https://github.com/XavierXiao/Dreambooth-Stable-Diffusion","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/XavierXiao%2FDreambooth-Stable-Diffusion","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/XavierXiao%2FDreambooth-Stable-Diffusion/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/XavierXiao%2FDreambooth-Stable-Diffusion/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/XavierXiao%2FDreambooth-Stable-Diffusion/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/XavierXiao","download_url":"https://codeload.github.com/XavierXiao/Dreambooth-Stable-Diffusion/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243618632,"owners_count":20320268,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["pytorch","pytorch-lightning","stable-diffusion","text-to-image"],"created_at":"2024-07-30T20:01:13.895Z","updated_at":"2025-03-14T17:30:52.330Z","avatar_url":"https://github.com/XavierXiao.png","language":"Jupyter Notebook","readme":"# Dreambooth on Stable Diffusion\n\nThis is an implementtaion of Google's [Dreambooth](https://arxiv.org/abs/2208.12242) with [Stable Diffusion](https://github.com/CompVis/stable-diffusion). The original Dreambooth is based on [Imagen](https://imagen.research.google/) text-to-image model. However, neither the model nor the pre-trained weights of Imagen is available. To enable people to fine-tune a text-to-image model with a few examples, I implemented the idea of Dreambooth on Stable diffusion.\n\nThis code repository is based on that of [Textual Inversion](https://github.com/rinongal/textual_inversion). Note that Textual Inversion only optimizes word ebedding, while dreambooth fine-tunes the whole diffusion model.\n\nThe implementation makes minimum changes over the official codebase of Textual Inversion. In fact, due to lazyness, some components in Textual Inversion, such as the embedding manager, are not deleted, although they will never be used here.\n## Update\n**9/20/2022**: I just found a way to reduce the GPU memory a bit. Remember that this code is based on Textual Inversion, and TI's code base has [this line](https://github.com/rinongal/textual_inversion/blob/main/ldm/modules/diffusionmodules/util.py#L112), which disable gradient checkpointing in a hard-code way. This is because in TI, the Unet is not optimized. However, in Dreambooth we optimize the Unet, so we can turn on the gradient checkpoint pointing trick, as in the original SD repo [here](https://github.com/CompVis/stable-diffusion/blob/main/ldm/modules/diffusionmodules/util.py#L112). The gradient checkpoint is default to be True in [config](https://github.com/XavierXiao/Dreambooth-Stable-Diffusion/blob/main/configs/stable-diffusion/v1-finetune_unfrozen.yaml#L47). I have updated the codes.\n## Usage\n\n### Preparation\nFirst set-up the ```ldm``` enviroment following the instruction from textual inversion repo, or the original Stable Diffusion repo.\n\nTo fine-tune a stable diffusion model, you need to obtain the pre-trained stable diffusion models following their [instructions](https://github.com/CompVis/stable-diffusion#stable-diffusion-v1). Weights can be downloaded on [HuggingFace](https://huggingface.co/CompVis). You can decide which version of checkpoint to use, but I use ```sd-v1-4-full-ema.ckpt```.\n\nWe also need to create a set of images for regularization, as the fine-tuning algorithm of Dreambooth requires that. Details of the algorithm can be found in the paper. Note that in the original paper, the regularization images seem to be generated on-the-fly. However, here I generated a set of regularization images before the training. The text prompt for generating regularization images can be ```photo of a \u003cclass\u003e```, where ```\u003cclass\u003e``` is a word that describes the class of your object, such as ```dog```. The command is\n\n```\npython scripts/stable_txt2img.py --ddim_eta 0.0 --n_samples 8 --n_iter 1 --scale 10.0 --ddim_steps 50  --ckpt /path/to/original/stable-diffusion/sd-v1-4-full-ema.ckpt --prompt \"a photo of a \u003cclass\u003e\" \n```\n\nI generate 8 images for regularization, but more regularization images may lead to stronger regularization and better editability. After that, save the generated images (separately, one image per ```.png``` file) at ```/root/to/regularization/images```.\n\n**Updates on 9/9**\nWe should definitely use more images for regularization. Please try 100 or 200, to better align with the original paper. To acomodate this, I shorten the \"repeat\" of reg dataset in the [config file](https://github.com/XavierXiao/Dreambooth-Stable-Diffusion/blob/main/configs/stable-diffusion/v1-finetune_unfrozen.yaml#L96).\n\nFor some cases, if the generated regularization images are highly unrealistic (happens when you want to generate \"man\" or \"woman\"), you can find a diverse set of images (of man/woman) online, and use them as regularization images.\n\n### Training\nTraining can be done by running the following command\n\n```\npython main.py --base configs/stable-diffusion/v1-finetune_unfrozen.yaml \n                -t \n                --actual_resume /path/to/original/stable-diffusion/sd-v1-4-full-ema.ckpt  \n                -n \u003cjob name\u003e \n                --gpus 0, \n                --data_root /root/to/training/images \n                --reg_data_root /root/to/regularization/images \n                --class_word \u003cxxx\u003e\n```\n\nDetailed configuration can be found in ```configs/stable-diffusion/v1-finetune_unfrozen.yaml```. In particular, the default learning rate is ```1.0e-6``` as I found the ```1.0e-5``` in the Dreambooth paper leads to poor editability. The parameter ```reg_weight``` corresponds to the weight of regularization in the Dreambooth paper, and the default is set to ```1.0```.\n\nDreambooth requires a placeholder word ```[V]```, called identifier, as in the paper. This identifier needs to be a relatively rare tokens in the vocabulary. The original paper approaches this by using a rare word in T5-XXL tokenizer. For simplicity, here I just use a random word ```sks``` and hard coded it.. If you want to change that, simply make a change in [this file](https://github.com/XavierXiao/Dreambooth-Stable-Diffusion/blob/main/ldm/data/personalized.py#L10).\n\nTraining will be run for 800 steps, and two checkpoints will be saved at ```./logs/\u003cjob_name\u003e/checkpoints```, one at 500 steps and one at final step. Typically the one at 500 steps works well enough. I train the model use two A6000 GPUs and it takes ~15 mins.\n\n### Generation\nAfter training, personalized samples can be obtained by running the command\n\n```\npython scripts/stable_txt2img.py --ddim_eta 0.0 \n                                 --n_samples 8 \n                                 --n_iter 1 \n                                 --scale 10.0 \n                                 --ddim_steps 100  \n                                 --ckpt /path/to/saved/checkpoint/from/training\n                                 --prompt \"photo of a sks \u003cclass\u003e\" \n```\n\nIn particular, ```sks``` is the identifier, which should be replaced by your choice if you happen to change the identifier, and ```\u003cclass\u003e``` is the class word ```--class_word``` for training.\n\n## Results\nHere I show some qualitative results. The training images are obtained from the [issue](https://github.com/rinongal/textual_inversion/issues/8) in the Textual Inversion repository, and they are 3 images of a large trash container. Regularization images are generated by prompt ```photo of a container```. Regularization images are shown here:\n\n![](assets/a-container-0038.jpg)\n\nAfter training, generated images with prompt ```photo of a sks container```:\n\n![](assets/photo-of-a-sks-container-0018.jpg)\n\nGenerated images with prompt ```photo of a sks container on the beach```:\n\n![](assets/photo-of-a-sks-container-on-the-beach-0017.jpg)\n\nGenerated images with prompt ```photo of a sks container on the moon```:\n\n![](assets/photo-of-a-sks-container-on-the-moon-0016.jpg)\n\nSome not-so-perfect but still interesting results:\n\nGenerated images with prompt ```photo of a red sks container```:\n\n![](assets/a-red-sks-container-0021.jpg)\n\nGenerated images with prompt ```a dog on top of sks container```:\n\n![](assets/a-dog-on-top-of-sks-container-0023.jpg)\n\n","funding_links":[],"categories":["Jupyter Notebook","New Concept Learning","Training","Papers","👑Stable Diffusion","图像生成"],"sub_categories":["Potentially Stale/Less active branches","Text-Image Generation","Python","资源传输下载"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FXavierXiao%2FDreambooth-Stable-Diffusion","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FXavierXiao%2FDreambooth-Stable-Diffusion","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FXavierXiao%2FDreambooth-Stable-Diffusion/lists"}