{"id":23885590,"url":"https://github.com/cai991108/sd3m-simpletuner-lora","last_synced_at":"2026-04-14T18:32:17.705Z","repository":{"id":270849298,"uuid":"911642668","full_name":"CAI991108/SD3M-SimpleTuner-lora","owner":"CAI991108","description":"This repository contains the code and resources for the project \"Fine-Tuning Stable Diffusion 3 Medium with SimpleTuner\", which focuses on enhancing generative art by fine-tuning the Stable Diffusion 3 Medium (SD3M) model using the SimpleTuner toolkit.","archived":false,"fork":false,"pushed_at":"2025-01-04T05:57:06.000Z","size":17998,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-23T02:44:49.440Z","etag":null,"topics":["fine-tuning","generative-art","lora","sd3","stability-ai","stable-diffusion","stable-diffusion-3","text2image"],"latest_commit_sha":null,"homepage":"https://huggingface.co/jimchoi/simpletuner-lora","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CAI991108.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-03T13:57:27.000Z","updated_at":"2025-01-16T08:12:48.000Z","dependencies_parsed_at":"2025-01-03T15:33:46.612Z","dependency_job_id":null,"html_url":"https://github.com/CAI991108/SD3M-SimpleTuner-lora","commit_stats":null,"previous_names":["cai991108/sd3m-simpletuner-lora"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CAI991108%2FSD3M-SimpleTuner-lora","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CAI991108%2FSD3M-SimpleTuner-lora/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CAI991108%2FSD3M-SimpleTuner-lora/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CAI991108%2FSD3M-SimpleTuner-lora/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CAI991108","download_url":"https://codeload.github.com/CAI991108/SD3M-SimpleTuner-lora/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240261497,"owners_count":19773474,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fine-tuning","generative-art","lora","sd3","stability-ai","stable-diffusion","stable-diffusion-3","text2image"],"created_at":"2025-01-04T05:44:29.897Z","updated_at":"2026-04-14T18:32:17.675Z","avatar_url":"https://github.com/CAI991108.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Fine-Tuning Stable Diffusion 3 Medium with SimpleTuner\n\n## Project Overview\n\nThis project explores the potential of fine-tuning the **Stable Diffusion 3 Medium (SD3M)** model using the **SimpleTuner** toolkit to generate higher quality and more diverse 2D concept art. The research focuses on understanding the technical differences between fine-tuning SD1.5/SDXL and SD3M, and leverages these insights to develop a robust fine-tuning methodology that can be applied across various artistic domains.\n\n## Key Features\n\n- **Fine-Tuning SD3M**: We fine-tune the SD3M model using the SimpleTuner toolkit, which simplifies the process of customizing and fine-tuning pre-trained models.\n- **Low Rank Adaptation (LoRA)**: We employ LoRA, a parameter-efficient fine-tuning technique, to adapt the model with limited computational resources.\n- **Evaluation of Generative Art**: We evaluate the fine-tuned model's ability to generate high-quality and diverse concept art, comparing it with the baseline SD3M model.\n\n## Installation\n\n### Prerequisites\n\n- Python 3.10 or 3.11\n- Poetry (for dependency management)\n\n### Setup\nThe [SimpleTuner SD3 Quickstart Guide](https://github.com/bghira/SimpleTuner/blob/main/documentation/quickstart/SD3.md)\nprovides a detailed guide on setting up the environment and fine-tuning the SD3M model using SimpleTuner.\n\nAdditionally, you might find the following instructions helpful:\n1. There are built-in scripts to install the dependencies and set up the environment. (see files `poetry.lock` and `pyproject.toml`).\nThe following `tool.poetry.source` will accelerate the installation process for most cases. **Or simply use the file provided in this repository**.\n   ```bash\n    name = \"mirrors\"\n    url = \"https://to/your/mirror\"\n    priority = \"primary\"\n   ```\n2. Choose wisely for the network connection. The suggestion is to `export` the following environment variables:\n   ```bash\n   export HF_ENDPOINT=https://huggingface.com\n   export WANDB_API_KEY=your_wandb_api_key_here\n   ```\n3. The experimental script `configure.py` provides interactive step-by-step configuration. However, one should manually modify the `multibackend.json`:\n    ```bash\n    \"cache_dir_vae\": \"cache/vae/sd3/xxx\",\n    \"instance_data_dir\": \"path/to/your/data\",\n    \"cache_dir\": \"/cache/text/sd3/xxx\",\n    ```\n\n## Example Usage of Generative Concept Art\n\n### John Singer Sargent Style Portrait\nAn example fine-tuned model mincing the art style of John Singer Sargent. The model was trained on a dataset of John Singer Sargent's portraits and can generate images in a similar style.\nplease refer to my HuggingFace model card for detail implementation  [jimchoi/simpletuner-lora](https://huggingface.co/jimchoi/simpletuner-lora)\n\n1. **Run the inference script**:\n   ```bash\n   python image_finetune.py --prompt \"your_prompt_here\"\n   ```\n\n2. **Adjust the prompt**:\n   - For better results, adjust the prompt content to avoid overfitting or generating overly similar images.\n\n## Research Limitations\n\n- **Dataset Size and Diversity**: The dataset used for fine-tuning is relatively small, which may limit the model's generalizability.\n- **Computational Resources**: Limited GPU resources constrained the batch size and training speed.\n- **Evaluation Techniques**: The evaluation is primarily based on visual inspection, and more rigorous quantitative metrics could be applied.\n\n## Future Work\n\n- **Expand Dataset**: Use a larger and more diverse dataset to improve the model's generalization capabilities.\n- **Multi-GPU Training**: Utilize multi-GPU environments to speed up the training process and handle larger datasets.\n- **Quantitative Evaluation**: Implement more rigorous quantitative evaluation metrics for a comprehensive assessment of the model's performance.\n\n## License\nThe project is licensed under the MIT License. See the [LICENSE](LICENSE) file for more details.\n\n## Acknowledgments\n\n- **Stability AI** for open-sourcing the SD3M model. The pre-trained SD3M model is available on Hugging Face: [stabilityai/stable-diffusion-3-medium-diffusers](https://huggingface.co/stabilityai/stable-diffusion-3-medium-diffusers).\n- **SimpleTuner** for providing an efficient fine-tuning toolkit. For more details on using SimpleTuner with SD3M, refer to the [SimpleTuner SD3 Quickstart Guide](https://github.com/bghira/SimpleTuner/blob/main/documentation/quickstart/SD3.md).\n\n---\n\nFor more details, please refer to the [report](report.pdf) and the Hugging Face model card [jimchoi/simpletuner-lora](https://huggingface.co/jimchoi/simpletuner-lora).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcai991108%2Fsd3m-simpletuner-lora","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcai991108%2Fsd3m-simpletuner-lora","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcai991108%2Fsd3m-simpletuner-lora/lists"}