{"id":13624995,"url":"https://github.com/TencentQQGYLab/ComfyUI-ELLA","last_synced_at":"2025-04-16T01:33:12.991Z","repository":{"id":234401482,"uuid":"788751269","full_name":"TencentQQGYLab/ComfyUI-ELLA","owner":"TencentQQGYLab","description":"ELLA nodes for ComfyUI","archived":false,"fork":false,"pushed_at":"2024-08-16T11:21:10.000Z","size":11428,"stargazers_count":341,"open_issues_count":30,"forks_count":17,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-11-08T13:38:57.750Z","etag":null,"topics":["comfy","comfyui","ella","stable-diffusion"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TencentQQGYLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-19T02:29:20.000Z","updated_at":"2024-11-04T12:17:36.000Z","dependencies_parsed_at":"2024-11-08T13:48:34.406Z","dependency_job_id":null,"html_url":"https://github.com/TencentQQGYLab/ComfyUI-ELLA","commit_stats":null,"previous_names":["tencentqqgylab/comfyui-ella"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TencentQQGYLab%2FComfyUI-ELLA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TencentQQGYLab%2FComfyUI-ELLA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TencentQQGYLab%2FComfyUI-ELLA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TencentQQGYLab%2FComfyUI-ELLA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TencentQQGYLab","download_url":"https://codeload.github.com/TencentQQGYLab/ComfyUI-ELLA/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249182497,"owners_count":21226077,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["comfy","comfyui","ella","stable-diffusion"],"created_at":"2024-08-01T21:01:49.419Z","updated_at":"2025-04-16T01:33:12.985Z","avatar_url":"https://github.com/TencentQQGYLab.png","language":"Python","funding_links":[],"categories":["Python","Workflows (3395) sorted by GitHub Stars","All Workflows Sorted by GitHub Stars"],"sub_categories":[],"readme":"# ComfyUI-ELLA\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"./assets/ELLA-Diffusion.jpg\" width=\"30%\" \u003e \u003cbr/\u003e\n\u003ca href='https://ella-diffusion.github.io/'\u003e\u003cimg src='https://img.shields.io/badge/Project-Page-green'\u003e\u003c/a\u003e\n\u003ca href='https://arxiv.org/abs/2403.05135'\u003e\u003cimg src='https://img.shields.io/badge/arXiv-2403.05135-b31b1b.svg'\u003e\u003c/a\u003e\n\u003c/div\u003e\n\n\n[ComfyUI](https://github.com/comfyanonymous/ComfyUI)  implementation for [ELLA](https://github.com/TencentQQGYLab/ELLA).\n\n## :star2: Changelog\n- **[2024.4.30]** Add a new node `ELLA Text Encode` to automatically concat ella and clip condition.\n- **[2024.4.24]** Upgraded ELLA Apply method. Better compatibility with the comfyui ecosystem. Refer to the method mentioned in [ComfyUI_ELLA PR #25](https://github.com/ExponentialML/ComfyUI_ELLA/pull/25)\n  - **DEPRECATED**: `Apply ELLA` without `simgas` is deprecated and it will be removed in a future version.\n- **[2024.4.22]** Fix unstable quality of image while multi-batch. Add CLIP concat (support lora trigger words now).\n- **[2024.4.19]** Documenting nodes.\n- **[2024.4.19]** Initial repo.\n\n## :pushpin: Notice\n\n- SIGMAS from node `BasicScheduler` or TIMESTEPS by node `Set ELLA Timesteps` must be the same as the KSampler settings. Because [Timestep-Aware Semantic Connector (TSC)](https://arxiv.org/html/2403.05135v1#S3), which dynamically adapts semantics features over sampling time steps, has been introduced.\n- If you need concat clip `CONDITIONING` to make LoRA trigger words effective, ELLA output `CONDITIONING` **always** needs to be linked to the `conditioning_to` of `Conditioning (Concat)` node.\n\n## :books: Example workflows\n\nThe [examples directory](./examples/) has workflow examples. You can directly load these images as workflow into ComfyUI for use.\n\n![workflow_example](./examples/workflow_example.png)\n\nAll legacy workflows was compatible. But it is deprecated and will be removed in a future version.\n\n![workflow_example_legacy](./examples/workflow_example_legacy.png)\n\n:tada: It works with controlnet! \n\n![workflow_controlnet](./examples/workflow_controlnet.png)\n\n:tada: It works with **lora trigger words** by concat CLIP CONDITIONING!\n\n:warning: NOTE again that `ELLA CONDITIONING` always needs to be linked to the `conditioning_to` of `Conditioning (Concat)` node.\n\n![workflow_lora](./examples/workflow_lora.png)\n\nWith `ELLA Text Encode` node, can simplify the workflow.\n\n![](./examples/workflow_ella_text_encode.png)\n\nWith the upgrade(2024.4.24), some interesting workflow can be implemented, such as using ELLA only in positive. As shown below:\n\n![workflow_lora_positive_ella_only](./examples/workflow_lora_positive_ella_only.png)\n\n| positive + negative | positive only |\n| :---: | :---: |\n| ![ella_with_lora](./assets/ella_with_lora.png) | ![ella_with_lora_positive_only](./assets/ella_with_lora_positive_only.png) |\n\nHowever, there is no guarantee that positive-only will bring better results.\n\nWorkflow with [AYS](https://research.nvidia.com/labs/toronto-ai/AlignYourSteps/).\n\n![workflow_ella_ays](./examples/workflow_ella_ays.png)\n\nAYS got with more visual details and better text-alignment, ref to [paper](https://research.nvidia.com/labs/toronto-ai/AlignYourSteps/).\n\n| w/ AYS | w/o AYS |\n| :---: | :---: |\n| ![](./assets/AYS_output.png) | ![](./assets/wo_AYS_output.png) |\n\nAnd [EMMA](https://github.com/TencentQQGYLab/ELLA/issues/15) is working in progress.\n\n## :green_book: Install\n\nDownload or git clone this repository inside ComfyUI/custom_nodes/ directory. `ComfyUI-ELLA` requires the latest version of ComfyUI. If something doesn't work be sure to upgrade.\n\n```bash\ncd ComfyUI/custom_nodes\ngit clone https://github.com/TencentQQGYLab/ComfyUI-ELLA\n```\n\nNext install dependencies.\n\n```bash\ncd ComfyUI-ELLA\npip install -r requirements.txt\n```\n\n## :orange_book: Models\n\nThese models must be placed in the corresponding directories under models.\n\nRemember you can also use any custom location setting an `ella` \u0026 `ella_encoder` entry in the `extra_model_paths.yaml` file.\n\n- `ComfyUI/models/ella`, create it if not present.\n  - Place [ELLA Models](https://huggingface.co/QQGYLab/ELLA) here\n- `ComfyUI/models/ella_encoder`, create it if not present.\n  - Place [FLAN-T5 XL Text Encoder](https://huggingface.co/QQGYLab/ELLA/tree/main/models--google--flan-t5-xl--text_encoder) here, it should be a folder of transfomers structure with config.json\n\nIn summary, you should have the following model directory structure:\n\n```bash\nComfyUI/models/ella/\n└── ella-sd1.5-tsc-t5xl.safetensors\n\nComfyUI/models/ella_encoder/\n└── models--google--flan-t5-xl--text_encoder\n    ├── config.json\n    ├── model.safetensors\n    ├── special_tokens_map.json\n    ├── spiece.model\n    ├── tokenizer_config.json\n    └── tokenizer.json\n```\n\n\n## :book: Nodes reference\n\n[Nodes reference](./NODES.md)\n\n## :mag: Common promblem\n\n- XXX not implemented for 'Half'. See [issue #12](https://github.com/TencentQQGYLab/ComfyUI-ELLA/issues/12#issuecomment-2067994702)\n- AYS + Ella getting dark image generations. See [issue #39](https://github.com/TencentQQGYLab/ComfyUI-ELLA/issues/39#issuecomment-2097354330)\n  - Check if `add_noise` of `SamplerCustom` node is enabled.\n  - Lower the `cfg` of `SamplerCustom` node.\n\n## :memo: TODO\n\n- [ ] Support prompt weighting\n\n## :hugs: Contributors (direct \u0026 indirect)\n\n\u003ctable\u003e\n\u003ctr\u003e\n   \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/JettHu\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/35261585?s=460\u0026amp;v=4\" width=\"32px;\" alt=\"\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eJettHu\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003c/td\u003e\n   \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/budui\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/16448529?s=460\u0026amp;v=4\" width=\"32px;\" alt=\"\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003ebudui\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003c/td\u003e\n   \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/kijai\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/40791699?s=460\u0026amp;v=4\" width=\"32px;\" alt=\"\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003ekijai\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003c/td\u003e\n   \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/huagetai\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/1137341?s=460\u0026amp;v=4\" width=\"32px;\" alt=\"\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003ehuagetai\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n## :yum: Thanks\n\n- ComfyUI: https://github.com/comfyanonymous/ComfyUI\n- Diffusers (borrowed timestep modules): https://github.com/huggingface/diffusers\n\n## :wink: Citation\n\n```\n@misc{hu2024ella,\n      title={ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment}, \n      author={Xiwei Hu and Rui Wang and Yixiao Fang and Bin Fu and Pei Cheng and Gang Yu},\n      year={2024},\n      eprint={2403.05135},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FTencentQQGYLab%2FComfyUI-ELLA","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FTencentQQGYLab%2FComfyUI-ELLA","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FTencentQQGYLab%2FComfyUI-ELLA/lists"}