{"id":23885589,"url":"https://github.com/dkackman/diffusers-workflow","last_synced_at":"2025-04-10T01:10:56.097Z","repository":{"id":263054335,"uuid":"889186815","full_name":"dkackman/diffusers-workflow","owner":"dkackman","description":"helper for the Huggingface Diffuser project","archived":false,"fork":false,"pushed_at":"2025-03-09T11:59:44.000Z","size":1845,"stargazers_count":5,"open_issues_count":6,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-10T01:10:47.151Z","etag":null,"topics":["diffusers","stable-diffusion","workflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dkackman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-15T19:27:20.000Z","updated_at":"2025-03-11T12:44:11.000Z","dependencies_parsed_at":"2024-11-15T21:26:48.880Z","dependency_job_id":"3f4f2ddd-7ccb-4a41-8b28-b1926fe6a22e","html_url":"https://github.com/dkackman/diffusers-workflow","commit_stats":null,"previous_names":["dkackman/diffusers-helper","dkackman/diffusers-workflow"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkackman%2Fdiffusers-workflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkackman%2Fdiffusers-workflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkackman%2Fdiffusers-workflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkackman%2Fdiffusers-workflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dkackman","download_url":"https://codeload.github.com/dkackman/diffusers-workflow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248137886,"owners_count":21053775,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["diffusers","stable-diffusion","workflow"],"created_at":"2025-01-04T05:44:29.443Z","updated_at":"2025-04-10T01:10:55.958Z","avatar_url":"https://github.com/dkackman.png","language":"Python","readme":"[![CodeQL](https://github.com/dkackman/diffusers-helper/actions/workflows/github-code-scanning/codeql/badge.svg)](https://github.com/dkackman/diffusers-helper/actions/workflows/github-code-scanning/codeql)\n\n# diffusers-workflow\n\n## Introduction\n\nThis is a simple, declaritive workflow engine for the [Huggingface Diffuser project](https://github.com/huggingface/diffusers). This command-line tool simplifies working with the Hugging Face Diffusers library by providing a flexible, JSON-based interface for running generative AI models. Users can define and execute complex image generation tasks without writing custom Python code.\n\n*Please [refer to the wiki](https://github.com/dkackman/diffusers-workflow/wiki) for more detailed instuctions.*\n\n## Features\n\n- Make any workflow command line executable with variable substitution\n- Suppport for text to image \u0026 video and image to image \u0026 video workflows\n- Image describing and prompt augmentation using locally installed LLMs\n- Image processing tasks for controlnet workflows\n- Composable workflows from multiple files\n- Other helpful tasks like background removal, upscaling, cropping and qr code generation\n\n## Installation\n\n### bash\n\n```bash\nbash ./install.sh\n. ./activate\npython -m dw.test\n```\n\n### powershell\n\n```powershell\n.\\install.ps1\n.\\venv\\scripts\\activate \npython -m dw.test\n```\n\n### Install Diffusers from Source\n\nThe install script will install [the diffusers library from PyPi](https://pypi.org/project/diffusers/). If you want to install from source and use not yet released diffusers, you can do so with the following commands:\n\n```bash\n. ./activate # or .\\venv\\scripts\\activate on windows\npip install git+https://github.com/huggingface/diffusers\n```\n\n### Run Tests\n\n```bash\n. ./activate # or .\\venv\\scripts\\activate on windows\npip install pytest\npytest -v\n```\n\n## Usage\n\n### Run a Workflow\n\n```bash\npython -m dw.run --help\nusage: run.py [-h] [-o OUTPUT_DIR] file_name [variables ...]\n\nRun a workflow from a file.\n\npositional arguments:\n  file_name             The filespec to of the workflow to run\n  variables             Optional parameters in name=value format\n\noptions:\n  -h, --help            show this help message and exit\n  -o OUTPUT_DIR, --output_dir OUTPUT_DIR\n                        The folder to write the outputs to\n```\n\n### Validate a Workflow Definition\n\n```bash\npython -m dw.validate --help\nusage: validate.py [-h] file_name\n\nValidate a project file.\n\npositional arguments:\n  file_name   The filespec to of the project to validate\n\noptions:\n  -h, --help  show this help message and exit\n```\n\n## JSON Input Format\n\n### Schema\n\n[Json schema](https://json-schema.app/view/%23?url=https%3A%2F%2Fraw.githubusercontent.com%2Fdkackman%2Fdiffusers-workflow%2Frefs%2Fheads%2Fmaster%2Fdw%2Fworkflow_schema.json)\n\n### Examples\n\n#### Simple Image Generation with an Input Variable\n\nThis example declares a variable for the `prompt` which can then be set on the command line. The `prompt` variable is then used in the `prompt` argument of the model.\n\n```bash\npython -m dw.run test_workflow.json prompt=\"an orange\" num_images_per_prompt=4\n```\n\n```json\n{\n    \"id\": \"test_workflow\",\n    \"variables\": {\n        \"prompt\": \"an apple\",\n        \"num_images_per_prompt\": 1\n    },\n    \"steps\": [\n        {\n            \"name\": \"main\",\n            \"pipeline\": {\n                \"configuration\": {\n                    \"component_type\": \"FluxPipeline\",\n                    \"offload\": \"sequential\"\n                },\n                \"from_pretrained_arguments\": {\n                    \"model_name\": \"black-forest-labs/FLUX.1-dev\",\n                    \"torch_dtype\": \"torch.bfloat16\"\n                },\n                \"arguments\": {\n                    \"prompt\": \"variable:prompt\",\n                    \"num_inference_steps\": 25,\n                    \"num_images_per_prompt\": \"variable:num_images_per_prompt\",\n                    \"guidance_scale\": 3.5,\n                    \"max_sequence_length\": 512\n                }\n            },\n            \"result\": {\n                \"content_type\": \"image/jpeg\"\n            }\n        }\n    ]\n}\n```\n\n#### Multiple Step Workflow\n\nThis example demonstrates a multiple step workflow including an image generation step followed by video generation. It includes the use of a transformer model for the image generation, a quantization example and an image to video model.\n\n```json\n{\n    \"id\": \"sd35_txt2img2vid\",\n    \"steps\": [\n        {\n            \"name\": \"image_generation\",\n            \"pipeline\": {\n                \"transformer\": {\n                    \"configuration\": {\n                        \"component_type\": \"SD3Transformer2DModel\",\n                        \"quantization_config\": {\n                            \"configuration\": {\n                                \"config_type\": \"BitsAndBytesConfig\"\n                            },\n                            \"arguments\": {\n                                \"load_in_4bit\": true,\n                                \"bnb_4bit_quant_type\": \"{nf4}\",\n                                \"bnb_4bit_compute_dtype\": \"torch.bfloat16\"\n                            }\n                        },\n                    },\n                    \"from_pretrained_arguments\": {\n                        \"model_name\": \"stabilityai/stable-diffusion-3.5-large\",\n                        \"subfolder\": \"transformer\",\n                        \"torch_dtype\": \"torch.bfloat16\"\n                    }\n                },\n                \"configuration\": {\n                    \"component_type\": \"StableDiffusion3Pipeline\",\n                    \"offload\": \"model\"\n                },\n                \"from_pretrained_arguments\": {\n                    \"model_name\": \"stabilityai/stable-diffusion-3.5-large\",\n                    \"torch_dtype\": \"torch.bfloat16\"\n                }\n                \"arguments\": {\n                    \"prompt\": \"portrait | wide angle shot of eyes off to one side of frame, lucid dream-like 3d model of owl, game asset, blender, looking off in distance ::8 style | glowing ::8 background | forest, vivid neon wonderland, particles, blue, green, orange ::7 parameters | rule of thirds, golden ratio, asymmetric composition, hyper- maximalist, octane render, photorealism, cinematic realism, unreal engine, 8k ::7 --ar 16:9 --s 1000\",\n                    \"num_inference_steps\": 25,\n                    \"guidance_scale\": 4.5,\n                    \"max_sequence_length\": 512\n                }\n            },\n            \"result\": {\n                \"content_type\": \"image/png\"\n            },\n        },\n        {\n            \"name\": \"image_to_video\",\n            \"pipeline\": {\n                \"configuration\": {\n                    \"offload\": \"sequential\",\n                    \"component_type\": \"CogVideoXImageToVideoPipeline\",\n                    \"vae\": {\n                        \"configuration\": {\n                            \"enable_slicing\": true,\n                            \"enable_tiling\": true\n                        }\n                    }\n                },\n                \"from_pretrained_arguments\": {\n                    \"model_name\": \"THUDM/CogVideoX-5b-I2V\",\n                    \"torch_dtype\": \"torch.bfloat16\"\n                },\n                \"arguments\": {\n                    \"image\": \"previous_result:image_generation\",\n                    \"prompt\": \"The owl stares intently and blinks\",\n                    \"num_videos_per_prompt\": 1,\n                    \"num_inference_steps\": 50,\n                    \"num_frames\": 49,\n                    \"guidance_scale\": 6\n                }\n            },\n            \"result\": {\n                \"content_type\": \"video/mp4\",\n                \"file_base_name\": \"owl\"\n            }\n        }\n    ]\n}\n```\n\n#### Prompt Augmentation\n\nThis example uses an instruct LLM to augment the prompt before passing it to the model. This also demonstrates composable child workflows.\n\nNote that this particular LLM requries `flash_attn` which in turn requires the [CUDA toolkit](https://developer.nvidia.com/cuda-toolkit).\n\n```json\n{\n    \"variables\": {\n        \"prompt\": \"A marmot, wearing a tophat in a woodland setting. Somewhat magical.\"\n    },\n    \"id\": \"sd35\",\n    \"steps\": [\n        {\n            \"name\": \"augment_prompt\",\n            \"workflow\": {\n                \"path\": \"builtin:augment_prompt.json\",\n                \"arguments\": {\n                    \"prompt\": \"variable:prompt\"\n                }\n            },\n            \"result\": {\n                \"content_type\": \"text/plain\"\n            }\n        },\n        {\n            \"name\": \"image_generation\",\n            \"pipeline\": {\n                \"configuration\": {\n                    \"component_type\": \"StableDiffusion3Pipeline\",\n                    \"offload\": \"sequential\"\n                },\n                \"from_pretrained_arguments\": {\n                    \"model_name\": \"stabilityai/stable-diffusion-3.5-large\",\n                    \"torch_dtype\": \"torch.bfloat16\"\n                },\n                \"arguments\": {\n                    \"prompt\": \"previous_result:augment_prompt\",\n                    \"num_inference_steps\": 25,\n                    \"guidance_scale\": 4.5,\n                    \"max_sequence_length\": 512,\n                    \"num_images_per_prompt\": 1\n                }\n            },\n            \"result\": {\n                \"content_type\": \"image/jpeg\"\n            }\n        }\n    ]\n}\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdkackman%2Fdiffusers-workflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdkackman%2Fdiffusers-workflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdkackman%2Fdiffusers-workflow/lists"}