{"id":13488234,"url":"https://github.com/mkshing/svdiff-pytorch","last_synced_at":"2026-01-14T20:40:04.207Z","repository":{"id":152047364,"uuid":"623291648","full_name":"mkshing/svdiff-pytorch","owner":"mkshing","description":"Implementation of \"SVDiff: Compact Parameter Space for Diffusion Fine-Tuning\"","archived":false,"fork":false,"pushed_at":"2024-01-24T23:35:09.000Z","size":14557,"stargazers_count":380,"open_issues_count":10,"forks_count":24,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-09-03T01:18:00.514Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2303.11305","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mkshing.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-04T04:40:12.000Z","updated_at":"2025-09-01T03:05:18.000Z","dependencies_parsed_at":null,"dependency_job_id":"fb82876c-0aee-4df8-9611-5c3dc1293331","html_url":"https://github.com/mkshing/svdiff-pytorch","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/mkshing/svdiff-pytorch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mkshing%2Fsvdiff-pytorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mkshing%2Fsvdiff-pytorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mkshing%2Fsvdiff-pytorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mkshing%2Fsvdiff-pytorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mkshing","download_url":"https://codeload.github.com/mkshing/svdiff-pytorch/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mkshing%2Fsvdiff-pytorch/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28434486,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T18:57:19.464Z","status":"ssl_error","status_checked_at":"2026-01-14T18:52:48.501Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T18:01:11.966Z","updated_at":"2026-01-14T20:40:04.181Z","avatar_url":"https://github.com/mkshing.png","language":"Jupyter Notebook","funding_links":[],"categories":["New Concept Learning","Jupyter Notebook"],"sub_categories":[],"readme":"# SVDiff-pytorch\n\u003ca href=\"https://colab.research.google.com/github/mkshing/svdiff-pytorch/blob/main/scripts/svdiff_pytorch.ipynb\" target=\"_parent\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/\u003e\u003c/a\u003e \n[![Open in Spaces](https://huggingface.co/datasets/huggingface/badges/raw/main/open-in-hf-spaces-sm.svg)](https://huggingface.co/spaces/svdiff-library/SVDiff-Training-UI)\n\n\nAn implementation of [SVDiff: Compact Parameter Space for Diffusion Fine-Tuning](https://arxiv.org/abs/2303.11305) by using d🧨ffusers. \n\nMy summary tweet is found [here](https://twitter.com/mk1stats/status/1642865505106272257).\n\n\n![result](assets/dog.png)\nleft: LoRA, right: SVDiff\n\n\nCompared with LoRA, the number of trainable parameters is 0.5 M less parameters and the file size is only 1.2MB (LoRA: 3.1MB)!!\n\n![kumamon](assets/kumamon.png)\n\n## Updates\n### 2023.4.11\n- Released v0.2.0 (please see [here](https://github.com/mkshing/svdiff-pytorch/releases/tag/v0.2.0) for the details). By this change, you get better results with less training steps than the first release v0.1.1!!\n- Add [Single Image Editing](#single-image-editing)\n  \u003cbr\u003e\n  ![chair-result](assets/chair-result.png)\n  \u003cbr\u003e\"photo of a ~~pink~~ **blue** chair with black legs\" (without DDIM Inversion) \n\n\n## Installation \n```\n$ pip install svdiff-pytorch\n```\nOr, manually \n```bash\n$ git clone https://github.com/mkshing/svdiff-pytorch\n$ pip install -r requirements.txt\n```\n\n## Single-Subject Generation\n\"Single-Subject Generation\" is a domain-tuning on a single object or concept (using 3-5 images). (See Section 4.1)\n\n### Training\nAccording to the paper, the learning rate for SVDiff needs to be 1000 times larger than the lr used for fine-tuning. \n\n```bash\nexport MODEL_NAME=\"runwayml/stable-diffusion-v1-5\"\nexport INSTANCE_DIR=\"path-to-instance-images\"\nexport CLASS_DIR=\"path-to-class-images\"\nexport OUTPUT_DIR=\"path-to-save-model\"\n\naccelerate launch train_svdiff.py \\\n  --pretrained_model_name_or_path=$MODEL_NAME  \\\n  --instance_data_dir=$INSTANCE_DIR \\\n  --class_data_dir=$CLASS_DIR \\\n  --output_dir=$OUTPUT_DIR \\\n  --with_prior_preservation --prior_loss_weight=1.0 \\\n  --instance_prompt=\"photo of a sks dog\" \\\n  --class_prompt=\"photo of a dog\" \\\n  --resolution=512 \\\n  --train_batch_size=1 \\\n  --gradient_accumulation_steps=1 \\\n  --learning_rate=1e-3 \\\n  --learning_rate_1d=1e-6 \\\n  --train_text_encoder \\\n  --lr_scheduler=\"constant\" \\\n  --lr_warmup_steps=0 \\\n  --num_class_images=200 \\\n  --max_train_steps=500\n```\n\n### Inference\n\n```python\nfrom diffusers import DiffusionPipeline, DPMSolverMultistepScheduler\nimport torch\n\nfrom svdiff_pytorch import load_unet_for_svdiff, load_text_encoder_for_svdiff\n\npretrained_model_name_or_path = \"runwayml/stable-diffusion-v1-5\"\nspectral_shifts_ckpt_dir = \"ckpt-dir-path\"\nunet = load_unet_for_svdiff(pretrained_model_name_or_path, spectral_shifts_ckpt=spectral_shifts_ckpt_dir, subfolder=\"unet\")\ntext_encoder = load_text_encoder_for_svdiff(pretrained_model_name_or_path, spectral_shifts_ckpt=spectral_shifts_ckpt_dir, subfolder=\"text_encoder\")\n# load pipe\npipe = StableDiffusionPipeline.from_pretrained(\n    pretrained_model_name_or_path,\n    unet=unet,\n    text_encoder=text_encoder,\n)\npipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)\npipe.to(\"cuda\")\nimage = pipe(\"A picture of a sks dog in a bucket\", num_inference_steps=25).images[0]\n```\n\nYou can use the following CLI too! Once it's done, you will see `grid.png` for the result.\n\n```bash\npython inference.py \\\n  --pretrained_model_name_or_path=\"runwayml/stable-diffusion-v1-5\"  \\\n  --spectral_shifts_ckpt=\"ckpt-dir-path\"  \\\n  --prompt=\"A picture of a sks dog in a bucket\"  \\\n  --scheduler_type=\"dpm_solver++\"  \\\n  --num_inference_steps=25  \\\n  --num_images_per_prompt=2 \n```\n\n## Single Image Editing\n### Training\nIn Single Image Editing, your instance prompt should be just the description of your input image **without the identifier**. \n\n```bash\nexport MODEL_NAME=\"runwayml/stable-diffusion-v1-5\"\nexport INSTANCE_DIR=\"dir-path-to-input-image\"\nexport CLASS_DIR=\"path-to-class-images\"\nexport OUTPUT_DIR=\"path-to-save-model\"\n\naccelerate launch train_svdiff.py \\\n  --pretrained_model_name_or_path=$MODEL_NAME  \\\n  --instance_data_dir=$INSTANCE_DIR \\\n  --class_data_dir=$CLASS_DIR \\\n  --output_dir=$OUTPUT_DIR \\\n  --instance_prompt=\"photo of a pink chair with black legs\" \\\n  --resolution=512 \\\n  --train_batch_size=1 \\\n  --gradient_accumulation_steps=1 \\\n  --learning_rate=1e-3 \\\n  --learning_rate_1d=1e-6 \\\n  --train_text_encoder \\\n  --lr_scheduler=\"constant\" \\\n  --lr_warmup_steps=0 \\\n  --max_train_steps=500\n```\n\n### Inference\n\n```python\nimport torch\nfrom PIL import Image\nfrom diffusers import DDIMScheduler\nfrom svdiff_pytorch import load_unet_for_svdiff, load_text_encoder_for_svdiff, StableDiffusionPipelineWithDDIMInversion\n\npretrained_model_name_or_path = \"runwayml/stable-diffusion-v1-5\"\nspectral_shifts_ckpt_dir = \"ckpt-dir-path\"\nimage = \"path-to-image\"\nsource_prompt = \"prompt-for-image\"\ntarget_prompt = \"prompt-you-want-to-generate\"\n\nunet = load_unet_for_svdiff(pretrained_model_name_or_path, spectral_shifts_ckpt=spectral_shifts_ckpt_dir, subfolder=\"unet\")\ntext_encoder = load_text_encoder_for_svdiff(pretrained_model_name_or_path, spectral_shifts_ckpt=spectral_shifts_ckpt_dir, subfolder=\"text_encoder\")\n# load pipe\npipe = StableDiffusionPipelineWithDDIMInversion.from_pretrained(\n    pretrained_model_name_or_path,\n    unet=unet,\n    text_encoder=text_encoder,\n)\npipe.scheduler = DDIMScheduler.from_config(pipe.scheduler.config)\npipe.to(\"cuda\")\n\n# (optional) ddim inversion\n# if you don't do it, inv_latents = None\nimage = Image.open(image).convert(\"RGB\").resize((512, 512))\n# in SVDiff, they use guidance scale=1 in ddim inversion\n# They use target_prompt in DDIM inversion for better results. See below for comparison between source_prompt and target_prompt.\ninv_latents = pipe.invert(target_prompt, image=image, guidance_scale=1.0).latents\n\n# They use a small cfg scale in Single Image Editing \nimage = pipe(target_prompt, latents=inv_latents, guidance_scale=3, eta=0.5).images[0]\n```\n\nDDIM inversion with target prompt (left) v.s. source prompt (right):\n\u003cbr\u003e\n![car-result](assets/car-result.png)\n\u003cbr\u003e\"photo of a grey ~~Beetle~~ **Mustang** car\" (original image: https://unsplash.com/photos/YEPDV3T8Vi8)\n\nTo use slerp to add more stochasticity,\n```python\nfrom svdiff_pytorch.utils import slerp_tensor\n\n# prev steps omitted\ninv_latents = pipe.invert(target_prompt, image=image, guidance_scale=1.0).latents\nnoise_latents = pipe.prepare_latents(inv_latents.shape[0], inv_latents.shape[1], 512, 512, dtype=inv_latents.dtype, device=pipe.device, generator=torch.Generator(\"cuda\").manual_seed(0))\ninv_latents =  slerp_tensor(0.5, inv_latents, noise_latents)\nimage = pipe(target_prompt, latents=inv_latents).images[0]\n```\n\n\n## Gradio\nYou can also try SVDiff-pytorch in a UI with [gradio](https://gradio.app/). This demo supports both training and inference!\n\n[![Open in Spaces](https://huggingface.co/datasets/huggingface/badges/raw/main/open-in-hf-spaces-sm.svg)](https://huggingface.co/spaces/svdiff-library/SVDiff-Training-UI)\n\nIf you want to run it locally, run the following commands step by step.\n```bash\n$ git clone --recursive https://github.com/mkshing/svdiff-pytorch.git\n$ cd scripts/gradio\n$ pip install -r requirements.txt \n$ export HF_TOKEN=\"YOUR_HF_TOKEN_HERE\"\n$ python app.py\n```\n\n## Additional Features\n\n### Spectral Shift Scaling\n\n![scale](assets/scale.png)\n\nYou can adjust the strength of the weights by `--spectral_shifts_scale`\n\nHere's a result for 0.8, 1.0, 1.2 (1.0 is the default).\n![scale-result](assets/scale-result.png)\n\n\n### Fast prior generation by using ToMe\nBy using [ToMe for SD](https://github.com/dbolya/tomesd), the prior generation can be faster! \n```\n$ pip install tomesd\n```\nAnd, add `--enable_tome_merging` to your training arguments!\n\n## Citation\n\n```bibtex\n@misc{https://doi.org/10.48550/arXiv.2303.11305,\n      title         = {SVDiff: Compact Parameter Space for Diffusion Fine-Tuning}, \n      author        = {Ligong Han and Yinxiao Li and Han Zhang and Peyman Milanfar and Dimitris Metaxas and Feng Yang},\n      year          = {2023},\n      eprint        = {2303.11305},\n      archivePrefix = {arXiv},\n      primaryClass  = {cs.CV},\n      url           = {https://arxiv.org/abs/2303.11305}\n}\n```\n\n```bibtex\n@misc{hu2021lora,\n      title         = {LoRA: Low-Rank Adaptation of Large Language Models},\n      author        = {Hu, Edward and Shen, Yelong and Wallis, Phil and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Lu and Chen, Weizhu},\n      year          = {2021},\n      eprint        = {2106.09685},\n      archivePrefix = {arXiv},\n      primaryClass  = {cs.CL}\n}\n```\n\n```bibtex\n@article{bolya2023tomesd,\n      title   = {Token Merging for Fast Stable Diffusion},\n      author  = {Bolya, Daniel and Hoffman, Judy},\n      journal = {arXiv},\n      url     = {https://arxiv.org/abs/2303.17604},\n      year    = {2023}\n}\n```\n\n## Reference\n- [DreamBooth in diffusers](https://github.com/huggingface/diffusers/tree/main/examples/dreambooth)\n- [DreamBooth in ShivamShrirao](https://github.com/ShivamShrirao/diffusers/tree/main/examples/dreambooth)\n- [Data from custom-diffusion](https://github.com/adobe-research/custom-diffusion#getting-started) \n\n## TODO\n- [x] Training\n- [x] Inference\n- [x] Scaling spectral shifts\n- [x] Support Single Image Editing\n- [ ] Support multiple spectral shifts (Section 3.2)\n- [ ] Cut-Mix-Unmix (Section 3.3)\n- [ ] SVDiff + LoRA\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmkshing%2Fsvdiff-pytorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmkshing%2Fsvdiff-pytorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmkshing%2Fsvdiff-pytorch/lists"}