{"id":13563136,"url":"https://github.com/cientgu/InstructDiffusion","last_synced_at":"2025-04-03T19:32:36.587Z","repository":{"id":193401616,"uuid":"688362518","full_name":"cientgu/InstructDiffusion","owner":"cientgu","description":"PyTorch implementation of InstructDiffusion, a unifying and generic framework for aligning computer vision tasks with human instructions.","archived":false,"fork":false,"pushed_at":"2024-05-14T08:47:40.000Z","size":15053,"stargazers_count":389,"open_issues_count":18,"forks_count":20,"subscribers_count":10,"default_branch":"main","last_synced_at":"2024-11-04T15:52:21.522Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cientgu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-07T07:36:51.000Z","updated_at":"2024-11-03T10:44:40.000Z","dependencies_parsed_at":"2024-01-14T03:48:18.232Z","dependency_job_id":"bb0451ce-e038-41a9-8118-8e2495d12b68","html_url":"https://github.com/cientgu/InstructDiffusion","commit_stats":null,"previous_names":["cientgu/instructdiffusion"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cientgu%2FInstructDiffusion","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cientgu%2FInstructDiffusion/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cientgu%2FInstructDiffusion/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cientgu%2FInstructDiffusion/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cientgu","download_url":"https://codeload.github.com/cientgu/InstructDiffusion/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247065434,"owners_count":20877777,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T13:01:15.512Z","updated_at":"2025-04-03T19:32:35.539Z","avatar_url":"https://github.com/cientgu.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# InstructDiffusion: A Generalist Modeling Interface for Vision Tasks\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://gengzigang.github.io/instructdiffusion.github.io/\"\u003eProject Page\u003c/a\u003e |\n  \u003ca href=\"https://arxiv.org/pdf/2309.03895.pdf\"\u003eArxiv\u003c/a\u003e |\n  \u003ca href=\"https://98cfca87ef8aed4a6b.gradio.live\"\u003eWeb Demo\u003c/a\u003e |\n  \u003ca href=\"#QuickStart\"\u003eQuickStart\u003c/a\u003e |\n  \u003ca href=\"#Training\"\u003eTraining\u003c/a\u003e |\n  \u003ca href=\"#Acknowledge\"\u003eAcknowledge\u003c/a\u003e |\n  \u003ca href='#Citation'\u003eCitation\u003c/a\u003e \n\u003c/p\u003e\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"figure/teaser.png\" width=\"1000\"/\u003e\n\u003c/div\u003e\n\nThis is the pytorch implementation of InstructDiffusion, a unifying and generic framework for aligning computer vision tasks with human instructions. Our code is based on the [Instruct-pix2pix](https://github.com/timothybrooks/instruct-pix2pix) and [CompVis/stable_diffusion](https://github.com/CompVis/stable-diffusion).\u003cbr\u003e\n\n## QuickStart\nFollow the steps below to quickly edit your own images. The inference code in our repository requires **one GPU with \u003e 9GB memory** to test images with a resolution of **512**.\n\n1. Clone this repo.\n2. Setup conda environment:\n   ```\n   conda env create -f environment.yaml\n   conda activate instructdiff\n   ```\n3. We provide a well-trained [checkpoint](https://mailustceducn-my.sharepoint.com/:u:/g/personal/aa397601_mail_ustc_edu_cn/EZmXduulFidIhJD73SGcbOoBNpm18CJmU4PgPTS21RM2Ow?e=KqQYpO) and a [checkpoint](https://mailustceducn-my.sharepoint.com/:u:/g/personal/aa397601_mail_ustc_edu_cn/EWlNmyeS9P1BkRg_IlXbPbwBeNMQXQTcIA0pCokyd61UWg?e=iKfRdk) that has undergone human-alignment. Feel free to download to the folder `checkpoints` and try both of them. Or you can download pre-trained models through `bash scripts/download_pretrained_instructdiffusion.sh`.\n\n4. You can edit your own images:\n```bash\npython edit_cli.py --input example.jpg --edit \"Transform it to van Gogh, starry night style.\"\n\n# Optionally, you can customize the parameters by using the following syntax: \n# --resolution 512 --steps 50 --config configs/instruct_diffusion.yaml --ckpt YOUR_CHECKPOINT --cfg-text 3.5 --cfg-image 1.25\n\n# We also support loading image from the website and edit, e.g., you could run the command like this:\npython edit_cli.py --input \"https://wallup.net/wp-content/uploads/2016/01/207131-animals-nature-lion.jpg\" \\\n   --edit \"Transform it to van Gogh, starry night style.\" \\\n   --resolution 512 --steps 50 \\\n   --config configs/instruct_diffusion.yaml \\\n   --ckpt checkpoints/v1-5-pruned-emaonly-adaption-task-humanalign.ckpt \\\n   --outdir logs/\n```\nFor other different tasks, we provide recommended parameter settings, which can be found in [`scripts/inference_example.sh`](./scripts/inference_example.sh).\n\n5. (Optional) You can launch your own interactive editing Gradio app:\n```bash\npython edit_app.py \n\n# You can also specify the path to the checkpoint\n# The default checkpoint is checkpoints/v1-5-pruned-emaonly-adaption-task-humanalign.ckpt\npython edit_app.py --ckpt checkpoints/v1-5-pruned-emaonly-adaption-task-humanalign.ckpt\n```\n\n## Training\nThe code is developed using python 3.8 on Ubuntu 18.04. The code is developed and tested using 48 NVIDIA V100 GPU cards, each with 32GB of memory. Other platforms are not fully tested.\n\n### Installation\n1. Clone this repo.\n2. Setup conda environment:\n   ```\n   conda env create -f environment.yaml\n   conda activate instructdiff\n   ```\n\n### Pre-trained Model Preparation\nYou can use the following command to download the official pre-trained stable diffusion model, or you can download the model trained by our pretraining adaptation process from [OneDrive](https://mailustceducn-my.sharepoint.com/:u:/g/personal/aa397601_mail_ustc_edu_cn/EXJSMIpFev5Nj0kuKI88U1IBZDSjegp3G8ukku0OxRRjFQ?e=QhnnB4) and put it into the following folder: stable_diffusion/models/ldm/stable-diffusion-v1/.\n   ```\n   bash scripts/download_pretrained_sd.sh\n   ```\n\n### Data Preparation\nYou can refer to the [dataset](https://github.com/cientgu/InstructDiffusion/tree/main/dataset) to prepare your data.\n\n### Training Command\nFor multi-GPU training on a single machine, you can use the following command:\n   ```\n   python -m torch.distributed.launch --nproc_per_node=8 main.py --name v0 --base configs/instruct_diffusion.yaml --train --logdir logs/instruct_diffusion\n   ```\n\nFor multi-GPU training on multiple machines, you can use the following command (assuming 6 machines as an example):\n   ```\n   bash run_multinode.sh instruct_diffusion v0 6\n   ```\n\n### Convert EMA-Model\nYou can get the final EMA checkpoint for inference using the command below:\n   ```\n   python convert_ckpt.py --ema-ckpt logs/instruct_diffusion/checkpoint/ckpt_epoch_200/state.pth --out-ckpt checkpoints/v1-5-pruned-emaonly-adaption-task.ckpt\n   ```\n\n## Acknowledge\n\nThanks to \n- [Stable-diffusion](https://github.com/CompVis/stable-diffusion)\n- [Instruct-pix2pix](https://github.com/timothybrooks/instruct-pix2pix)\n\n## Citation\n\n```\n@inproceedings{Geng23instructdiff,\n  author       = {Zigang Geng and\n                  Binxin Yang and\n                  Tiankai Hang and\n                  Chen Li and\n                  Shuyang Gu and\n                  Ting Zhang and\n                  Jianmin Bao and\n                  Zheng Zhang and\n                  Han Hu and\n                  Dong Chen and\n                  Baining Guo},\n  title        = {InstructDiffusion: {A} Generalist Modeling Interface for Vision Tasks},\n  booktitle = {Proc. CVPR},\n  year      = {2024},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcientgu%2FInstructDiffusion","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcientgu%2FInstructDiffusion","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcientgu%2FInstructDiffusion/lists"}