{"id":18573558,"url":"https://github.com/prs-eth/dginstyle","last_synced_at":"2025-04-09T16:05:19.625Z","repository":{"id":231189267,"uuid":"731352501","full_name":"prs-eth/DGInStyle","owner":"prs-eth","description":"[ECCV 2024] DGInStyle: Domain-Generalizable Semantic Segmentation with Image Diffusion Models and Stylized Semantic Control","archived":false,"fork":false,"pushed_at":"2024-11-24T19:28:40.000Z","size":3965,"stargazers_count":75,"open_issues_count":7,"forks_count":9,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-09T16:05:14.944Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/prs-eth.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-13T22:23:52.000Z","updated_at":"2025-04-06T08:59:46.000Z","dependencies_parsed_at":"2025-01-16T16:08:53.969Z","dependency_job_id":"b3c51de8-15d3-433e-9fd6-ab7ccb5d0ffb","html_url":"https://github.com/prs-eth/DGInStyle","commit_stats":null,"previous_names":["yurujaja/dginstyle"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prs-eth%2FDGInStyle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prs-eth%2FDGInStyle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prs-eth%2FDGInStyle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prs-eth%2FDGInStyle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/prs-eth","download_url":"https://codeload.github.com/prs-eth/DGInStyle/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248065289,"owners_count":21041871,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T23:10:51.276Z","updated_at":"2025-04-09T16:05:19.605Z","avatar_url":"https://github.com/prs-eth.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n## :minidisc: DGInStyle: Domain-Generalizable Semantic Segmentation with Image Diffusion Models and Stylized Semantic Control\n\n**ECCV 2024**\n[[Project Page]](https://dginstyle.github.io/) | [[ArXiv]](https://arxiv.org/abs/2312.03048) |\n[[Datasets]](https://drive.google.com/file/d/1e2wiwr5_wgCN3pLCQMOQvjRqQYtEozWy/view)\n\nBy [Yuru Jia](https://github.com/yurujaja),\n[Lukas Hoyer](https://lhoyer.github.io/),\n[Shengyu Huang](https://shengyuh.github.io/),\n[Tianfu Wang](https://tianfwang.github.io/),\n[Luc Van Gool](https://scholar.google.com/citations?user=TwMib_QAAAAJ\u0026hl=en),\n[Konrad Schindler](https://scholar.google.com/citations?user=FZuNgqIAAAAJ\u0026hl=en),\n[Anton Obukhov](https://www.obukhov.ai/)\n\nWe propose DGInStyle, an efficient data generation pipeline with a pretrained text-to-image latent diffusion model (LDM) at its core. The semantic segmentation models trained on our generated dataset offer improved domain generalization, drawing on the prior knowledge embedded in the LDM.\n\nThis Repository hosts the training and inference implementation for the **DGInStyle data generation pipeline**. To assess the effectiveness of the generated data, we provide [DGInStyle-SegModel](https://github.com/yurujaja/DGInStyle-SegModel), which is designed for downstream semantic segmentation tasks.\n\n\u003cimg src=\"figs/teaser_v02.jpg\" width=\"85%\" height=\"60%\"\u003e\n\n\u003c!-- ## :hourglass: ToDo\n:white_large_square: Release the guidance for DGInStyle pipeline training.\n\n:ballot_box_with_check: Release the code and weights for the semantic segmentation models.\n\n:ballot_box_with_check: Release demo script for fine-grained stylized conditioned image generation.\n\n:ballot_box_with_check: Release arXiv paper. --\u003e\n\n## :gear: Setup\nFor this project, we used python/3.8.5, cuda/12.1.1. \n\n### Repository\nClone the repository:\n\n```bash\ngit clone https://github.com/yurujaja/DGInStyle.git\ncd DGInStyle\n```\n\n### Dependencies\n```bash\nexport DGINSTYLE_PATH=/path/to/venv/dginstyle  # change this\npython3 -m venv ${DGINSTYLE_PATH}\nsource ${DGINSTYLE_PATH}/bin/activate\n\npip install -r requirements.txt\n```\n\n## :desktop_computer: Inference Demo\nWe provide a demo inference script given some example semantic conditions. The weights are available via [HuggingFace repository](https://huggingface.co/yurujaja/DGInStyle), including the source-domain data (GTA) fine-tuned Stable Diffusion weights, ControlNet weights and SegFormer weights trained using DGInStyle. For prior-domain data (LAION-5B) pretrained Stable Diffusion weights, we adopt [RunwayML's stable-diffusion-v1-5](https://huggingface.co/runwayml/stable-diffusion-v1-5).\n\n\u003c!-- To directly interact with the Hugging Face Hub, you need to have a Hugging Face account with your access token, then login. \n```bash\nhuggingface-cli login\n``` --\u003e\nWe provide `example_data` so that you can preview the data created under specific semantic conditions. Please run the command given below. The output will be saved by default in the `example_data/output` folder.\n```bash\npython demo.py\n```\n\n### Checkpoint cache\nThe weights are automatically downloaded to the Hugging Face cache by default. The location of this cache can be changed by overriding the HF_HOME environment variable:\n\n```bash\nexport HF_HOME=new/path   # change this\n```\n\n## :baseball: Training \n### Preparing for datasets\nFollowing the common practice in domain generalization, we use **GTA** as the synthetic source dataset: Please download all image and label packages from [here](https://download.visinf.tu-darmstadt.de/data/from_games/) and extract them to `data/gta`. Run the following script to get dataset training file:\n```bash\npython -m controlnet.tools.convert_dataset_file\n```\nTo enable rare class sampling (rcs) component, please refer to [this repository](https://github.com/yurujaja/DGInStyle-SegModel#:~:text=GTA%3A%20Please%20download%20all,py%20%20%20%23%20sample%20GTA%20subsets) to generate rcs related files.\n\u003c!-- To enable rare class sampling (rcs) component, generate rcs related files using the following script:\n```bash\npython -m \n``` --\u003e\n\n### DreamBooth fine-tuning using GTA\nAs a first step of our Style Swap technique, we fine-tune the original latent diffusion model's U-Net $\\mathbf{U}^{\\mathbf{P}}$ with the [Dreambooth protocol](https://dreambooth.github.io/) towards the source domain using all GTA images. \n```bash\npython dreambooth/train_dreambooth.py \\\n    --pretrained_model_name_or_path='runwayml/stable-diffusion-v1-5' \\\n    --instance_data_dir='data/gta/images/' \\\n    --output_dir='path/to/your/dreambooth/output' \\    # change this\n    --instance_prompt='' \\\n    --resolution=512 \\\n    --train_batch_size=1 \\\n    --sample_batch_size=1 \\\n    --gradient_accumulation_steps=2 \\\n    --gradient_checkpointing \\\n    --learning_rate=2e-06 \\\n    --lr_scheduler=constant \\\n    --lr_warmup_steps=1000 \\\n    --max_train_steps=30000 \\\n    --mixed_precision=fp16 \\\n    --report_to=wandb \\     # choose from wandb or tensorboard\n    --validation_prompt='roads building sky person vegetation car' \\  # change this\n    --validation_steps=200 \\\n    --checkpointing_steps=1000\n```\n\n### ControlNet training with the source domain LDM\nThe above resulting U-Net $\\mathbf{U}^{\\mathbf{S}}$ is used as the base model instead of $\\mathbf{U}^{\\mathbf{P}}$ to initialize ControlNet, and still the paired source domain data is used to let the ControlNet focus primarily on the task-specific yet style-agnostic layout control.\n```bash\nexport U_S_MODEL_DIR='path/to/your/dreambooth/output'  # change this\nexport OUTPUT_DIR='path/to/your/controlnet/output' # change this\nexport DATASET_FILE='data/gta/dataset_file.json'\n\naccelerate launch controlnet/train.py \\\n    --pretrained_model_name_or_path=$MODEL_DIR \\\n    --inference_basemodel_path='runwayml/stable-diffusion-v1-5' \\\n    --output_dir=$OUTPUT_DIR \\\n    --dataset_file=$DATASET_FILE \\\n    --learning_rate=1e-05 \\\n    --train_batch_size=2 \\\n    --gradient_accumulation_steps=4 \\\n    --checkpointing_steps=2000 \\\n    --validation_steps=1000 \\\n    --report_to='wandb' \\\n    --rcs_enabled \\\n    --rcs_data_root='data/gta/' \\\n    --tracker_project_name='dginstyle_training' \\\n    --validate_file='controlnet/tools/example_validation.jsonl'\n```\n\n\u003c!-- ## :floppy_disk: Generating Datasets --\u003e\n\n\n## :shamrock: Acknowledgements\n\nThis repository is based on the following open-source projects. We thank their authors for making the source code publicly available.\n\n* [Diffusers](https://github.com/huggingface/diffusers)\n* [ControlNet](https://github.com/lllyasviel/ControlNet)\n* [MultiDiffusion](https://github.com/omerbt/MultiDiffusion)\n* [DAFormer](https://github.com/lhoyer/DAFormer)\n\n\n## :framed_picture: License\n\nThis project is released under the [Apache License 2.0](LICENSE), while some specific features in this repository are with other licenses. Please refer to [LICENSE](LICENSE) for the careful check, if you are using our code for commercial matters.\n\n## :page_with_curl: Citation\n```bibtex\n@inproceedings{jia2024dginstyle,\n  title     = {DGInStyle: Domain-Generalizable Semantic Segmentation with Image Diffusion Models and Stylized Semantic Control}, \n  author    = {Yuru Jia and Lukas Hoyer and Shengyu Huang and Tianfu Wang and Luc Van Gool and Konrad Schindler and Anton Obukhov},\n  booktitle = {European Conference on Computer Vision},\n  year      = {2024},\n}\n```\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprs-eth%2Fdginstyle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprs-eth%2Fdginstyle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprs-eth%2Fdginstyle/lists"}