{"id":28532294,"url":"https://github.com/sail-sg/finetune-fair-diffusion","last_synced_at":"2025-07-18T06:35:58.773Z","repository":{"id":211100937,"uuid":"726783221","full_name":"sail-sg/finetune-fair-diffusion","owner":"sail-sg","description":"Code of the paper: Finetuning Text-to-Image Diffusion Models for Fairness","archived":false,"fork":false,"pushed_at":"2024-04-26T05:38:20.000Z","size":49365,"stargazers_count":43,"open_issues_count":1,"forks_count":3,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-07-07T14:41:27.387Z","etag":null,"topics":["diffusion-models","fairness","text-to-image","trustworthy-ai"],"latest_commit_sha":null,"homepage":"https://sail-sg.github.io/finetune-fair-diffusion/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sail-sg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-12-03T11:48:13.000Z","updated_at":"2025-05-31T23:53:28.000Z","dependencies_parsed_at":null,"dependency_job_id":"c5edd4b7-222e-4463-8383-852e4118a674","html_url":"https://github.com/sail-sg/finetune-fair-diffusion","commit_stats":null,"previous_names":["sail-sg/finetune-fair-diffusion"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sail-sg/finetune-fair-diffusion","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sail-sg%2Ffinetune-fair-diffusion","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sail-sg%2Ffinetune-fair-diffusion/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sail-sg%2Ffinetune-fair-diffusion/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sail-sg%2Ffinetune-fair-diffusion/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sail-sg","download_url":"https://codeload.github.com/sail-sg/finetune-fair-diffusion/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sail-sg%2Ffinetune-fair-diffusion/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265711103,"owners_count":23815471,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["diffusion-models","fairness","text-to-image","trustworthy-ai"],"created_at":"2025-06-09T15:38:04.440Z","updated_at":"2025-07-18T06:35:58.761Z","avatar_url":"https://github.com/sail-sg.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Finetuning Text-to-Image Diffusion Models for Fairness\nCode and checkpoints for the paper [Finetuning Text-to-Image Diffusion Models for Fairness](https://openreview.net/forum?id=hnrB5YHoYu) (ICLR 2024, oral presentation).\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"./_github-images/original-SD-prompt_3.jpg\" alt=\"Images generated by the original SD\" style=\"float: left; width: 48%;\" /\u003e\n  \u003cimg src=\"./_github-images/debiased-SD-prompt_3.jpg\" alt=\"Images generated by the debiased SD\" style=\"float: right; width:48%;\" /\u003e\n  \u003cdiv style=\"clear: both;\"\u003e\u003c/div\u003e\n\u003c/div\u003e \n\u003cp align=\"center\"\u003e\u003cem\u003ePrompt: \"A photo of the face of a electrical and electronics repairer, a person\". Left images are generated by the original stable-diffusion-v1-5. Right images are generated by the finetuned stable diffusion, debiased for gender and race jointly. For every image, the first color-coded bar denotes the predicted gender: male (\u003cspan style=\"color: blue;\"\u003eblue\u003c/span\u003e) or female (\u003cspan style=\"color: red;\"\u003ered\u003c/span\u003e). The second denotes race: WMELH (\u003cspan style=\"color: green;\"\u003egreen\u003c/span\u003e), Asian (\u003cspan style=\"color: orange;\"\u003eorange\u003c/span\u003e), Black (\u003cspan style=\"color: black;\"\u003eblack\u003c/span\u003e), or Indian (\u003cspan style=\"color: brown;\"\u003ebrown\u003c/span\u003e). WMELH is the broad race class of White, Middle Eastern, and Latino Hispanic. Bar height represents prediction confidence. Bounding boxes denote detected faces. Images with the same number label are generated using the same noise.\u003c/em\u003e\u003c/p\u003e\n\n## Prerequisite:\nThis project has been tested under the following environment:\n- OS: Red Hat Enterprise Linux 8.4\n- GPU: NVIDIA A100-SXM4-40GB ($\\times$ 2, or more)\n- CUDA Version: 12.2\n\n1. We manage environments with Conda. To set up the environment, follow these steps:\n```\nconda env create -f environment.yml\nconda activate fair-diffusion\n```\n\n2. To set up wandb for logging, run:\n```\nwandb login\n```\n\n3. Download data.zip from this [link](https://drive.google.com/file/d/18FAu5rxGC2AlC3IN43lukE6Rgyyff3AR/view?usp=share_link) to the working directory, unzip it by:\n```\nunzip data.zip\n```\nThis will create a data folder in the current working directory.\n\n\n## Experiments:\nThis repository includes the code and checkpoints necessary to replicate the five experiments detailed in our paper. To find more information on a specific experiment, please click on the experiment's name, which will direct you to the corresponding folder.\n| Experiment Name | Prompts | Description |\n|------------|---|-----------------|\n| [exp-1-debias-gender](exp-1-debias-gender/) | Occupations | Finetune LoRA on text encoder and/or U-Net to debias binary gender to a perfectly balanced distribtion. |\n| [exp-2-debias-gender-token](exp-2-debias-gender-token/) | Occupations | Finetune prompt prefix to debias binary gender to a perfectly balanced distribtion. |\n| [exp-3-debias-gender-race](exp-3-debias-gender-race) | Occupations | Finetune LoRA on text encoder and/or U-Net to jointly debias binary gender and four classes of race, to a perfectly balanced distribtion. |\n| [exp-4-debias-gender-race-age](exp-4-debias-gender-race-age) | Occupations | Finetune LoRA on text encoder and/or U-Net to jointly debias binary gender and four classes of race, to a perfectly balanced distribtion. Simultaneouly, control age to 75% yound and 25% old distribution. |\n| [exp-5-debias-gender-race-multi-concepts](exp-5-debias-gender-race-multi-concepts) | (1) Occupations, (2) occupations with style and context, (3) personal descriptors, and (4) sports  | Finetune LoRA on text encoder and/or U-Net to jointly debias binary gender and four classes of race, to a perfectly balanced distribtion. |\n| [exp-6-debias-race](exp-6-debias-race) | Occupations  | Finetune LoRA on text encoder and/or U-Net to debias four classes of race, to a perfectly balanced distribtion. |\n\n\n## Fair diffusion model adaptors\nIf you're primarily interested in utilizing and evaluating our debiased Stable Diffusion v1-5, please follow below instructions.\n\n\n| Experiment Name | Checkpoints | Instructions on how to use |\n|------------|---|-----------------|\n| [exp-1-debias-gender](exp-1-debias-gender/) | [link](https://drive.google.com/file/d/1G6sl8Hv8ZX1poaqTHiBGDdz1Sqj1bs4u/view?usp=share_link) | unzip to `exp-1-debias-gender/outputs`, follow instructions starting from [this point onwards](exp-1-debias-gender/README.md#step-3). |\n| [exp-2-debias-gender-token](exp-2-debias-gender-token/) | [link](https://drive.google.com/file/d/1jRgL5hCxeME5q2FOMUOmKbiGh0awKMq8/view?usp=share_link) | unzip to `exp-2-debias-gender-token/outputs`, follow instructions starting from [this point onwards](exp-2-debias-gender-token/README.md#step-3). |\n| [exp-3-debias-gender-race](exp-3-debias-gender-race) | [link](https://drive.google.com/file/d/1aYCBNnIJlj3nrIYd8o-D4ob02JLVQnpk/view?usp=share_link) | unzip to `exp-3-debias-gender-race/outputs`, follow instructions starting from [this point onwards](exp-3-debias-gender-race/README.md#step-3). |\n| [exp-4-debias-gender-race-age](exp-4-debias-gender-race-age) | [link](https://drive.google.com/file/d/1KSd3BE67xq3LONH7rcBk6c0qEvQlRhDQ/view?usp=share_link) | unzip to `exp-4-debias-gender-race-age/outputs`, follow instructions starting from [this point onwards](exp-4-debias-gender-race-age/README.md#step-3). |\n| [exp-5-debias-gender-race-multi-concepts](exp-5-debias-gender-race-multi-concepts) | [link](https://drive.google.com/file/d/1oV5WBl8DKFEHrI4fzBJLekK1GPyNKRpF/view?usp=share_link) | unzip to `exp-5-debias-gender-race-multi-concepts/outputs`, follow instructions starting from [this point onwards](exp-5-debias-gender-race-multi-concepts/README.md#step-3). |\n| [exp-6-debias-race](exp-6-debias-race) | [link](https://drive.google.com/file/d/1u1zy3Q4dINYU17YK3ckB3lMyrKe_20LA/view?usp=share_link) | unzip to `exp-6-debias-race/outputs`, follow instructions starting from [this point onwards](exp-6-debias-race/README.md#step-3). |\n\n\n# Get in touch\nWe sincerely appreciate your interest in our work. Should you have any questions, please don't hesitate to reach out to Xudong at xudong.shen@u.nus.edu.\n\n# Citation\n\n```\n@inproceedings{shen2024finetuning,\n      title={Finetuning Text-to-Image Diffusion Models for Fairness}, \n      author={Xudong Shen and Chao Du and Tianyu Pang and Min Lin and Yongkang Wong and Mohan Kankanhalli},\n      booktitle={The Twelfth International Conference on Learning Representations},\n      year={2024},\n      url={https://openreview.net/forum?id=hnrB5YHoYu}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsail-sg%2Ffinetune-fair-diffusion","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsail-sg%2Ffinetune-fair-diffusion","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsail-sg%2Ffinetune-fair-diffusion/lists"}