{"id":15030842,"url":"https://github.com/tencent/hunyuandit","last_synced_at":"2025-05-13T22:03:51.439Z","repository":{"id":239689027,"uuid":"798671457","full_name":"Tencent/HunyuanDiT","owner":"Tencent","description":"Hunyuan-DiT : A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding","archived":false,"fork":false,"pushed_at":"2025-01-13T03:22:41.000Z","size":120914,"stargazers_count":4045,"open_issues_count":122,"forks_count":336,"subscribers_count":42,"default_branch":"main","last_synced_at":"2025-04-10T15:57:15.695Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://dit.hunyuan.tencent.com/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Tencent.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-10T08:47:15.000Z","updated_at":"2025-04-10T11:48:55.000Z","dependencies_parsed_at":"2025-04-10T14:00:17.555Z","dependency_job_id":null,"html_url":"https://github.com/Tencent/HunyuanDiT","commit_stats":null,"previous_names":["tencent/hunyuandit"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tencent%2FHunyuanDiT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tencent%2FHunyuanDiT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tencent%2FHunyuanDiT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tencent%2FHunyuanDiT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Tencent","download_url":"https://codeload.github.com/Tencent/HunyuanDiT/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251338004,"owners_count":21573483,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-24T20:14:23.202Z","updated_at":"2025-04-28T15:28:22.811Z","avatar_url":"https://github.com/Tencent.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!-- ## **HunyuanDiT** --\u003e\r\n\r\n\u003cp align=\"center\"\u003e\r\n  \u003cimg src=\"https://raw.githubusercontent.com/Tencent/HunyuanDiT/main/asset/logo.png\"  height=100\u003e\r\n\u003c/p\u003e\r\n\r\n# Hunyuan-DiT : A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding\r\n\r\n\u003cdiv align=\"center\"\u003e\r\n  \u003ca href=\"https://github.com/Tencent/HunyuanDiT\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=Hunyuan-DiT Code\u0026message=Github\u0026color=blue\u0026logo=github-pages\"\u003e\u003c/a\u003e \u0026ensp;\r\n  \u003ca href=\"https://dit.hunyuan.tencent.com\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=Project%20Page\u0026message=Github\u0026color=blue\u0026logo=github-pages\"\u003e\u003c/a\u003e \u0026ensp;\r\n  \u003ca href=\"https://arxiv.org/abs/2405.08748\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=Tech Report\u0026message=Arxiv:HunYuan-DiT\u0026color=red\u0026logo=arxiv\"\u003e\u003c/a\u003e \u0026ensp;\r\n  \u003ca href=\"https://arxiv.org/abs/2403.08857\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=Paper\u0026message=Arxiv:DialogGen\u0026color=red\u0026logo=arxiv\"\u003e\u003c/a\u003e \u0026ensp;\r\n  \u003ca href=\"https://huggingface.co/Tencent-Hunyuan/HunyuanDiT\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=Hunyuan-DiT\u0026message=HuggingFace\u0026color=yellow\"\u003e\u003c/a\u003e \u0026ensp;\r\n  \u003ca href=\"https://hunyuan.tencent.com/bot/chat\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=Hunyuan Bot\u0026message=Web\u0026color=green\"\u003e\u003c/a\u003e \u0026ensp;\r\n  \u003ca href=\"https://huggingface.co/spaces/Tencent-Hunyuan/HunyuanDiT\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=Hunyuan-DiT Demo\u0026message=HuggingFace\u0026color=yellow\"\u003e\u003c/a\u003e \u0026ensp;\r\n\u003c/div\u003e\r\n\r\n-----\r\n\r\nThis repo contains PyTorch model definitions, pre-trained weights and inference/sampling code for our paper exploring Hunyuan-DiT. You can find more visualizations on our [project page](https://dit.hunyuan.tencent.com/).\r\n\r\n\u003e [**Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding**](https://arxiv.org/abs/2405.08748) \u003cbr\u003e\r\n\r\n\u003e [**DialogGen: Multi-modal Interactive Dialogue System for Multi-turn Text-to-Image Generation**](https://arxiv.org/abs/2403.08857) \u003cbr\u003e\r\n\r\n## 🔥🔥🔥 News!!\r\n* Jul 15, 2024: 🚀 HunYuanDiT and Shakker.Ai have jointly launched a fine-tuning event based on the HunYuanDiT 1.2 model. By publishing a lora or fine-tuned model based on HunYuanDiT, you can earn up to $230 bonus from Shakker.Ai. See [Shakker.Ai](https://www.shakker.ai/activitys/shaker-the-world-hunyuan) for more details.\r\n* Jul 15, 2024: :tada: Update ComfyUI to support standardized workflows and compatibility with weights from t2i module and Lora training for versions 1.1/1.2, as well as those trained by Kohya or the official script. See [ComfyUI](./comfyui-hydit) for details.\r\n* Jul 15, 2024: :zap: We offer Docker environments for CUDA 11/12, allowing you to bypass complex installations and play with a single click! See [dockers](#installation-guide-for-linux) for details. \r\n* Jul 08, 2024: :tada: HYDiT-v1.2 version is released. Please check [HunyuanDiT-v1.2](https://huggingface.co/Tencent-Hunyuan/HunyuanDiT-v1.2) and [Distillation-v1.2](https://huggingface.co/Tencent-Hunyuan/Distillation-v1.2) for more details.\r\n* Jul 03, 2024: :tada: Kohya-hydit version now available for v1.1 and v1.2 models, with GUI for inference. Official Kohya version is under review. See [kohya](./kohya_ss-hydit) for details.\r\n* Jun 27, 2024: :art: Hunyuan-Captioner is released, providing fine-grained caption for training data. See [mllm](./mllm) for details.\r\n* Jun 27, 2024: :tada: Support LoRa and ControlNet in diffusers. See [diffusers](./diffusers) for details.\r\n* Jun 27, 2024: :tada: 6GB GPU VRAM Inference scripts are released. See [lite](./lite) for details.\r\n* Jun 19, 2024: :tada: ControlNet is released, supporting canny, pose and depth control. See [training/inference codes](#controlnet) for details.\r\n* Jun 13, 2024: :zap: HYDiT-v1.1 version is released, which mitigates the issue of image oversaturation and alleviates the watermark issue. Please check [HunyuanDiT-v1.1](https://huggingface.co/Tencent-Hunyuan/HunyuanDiT-v1.1) and \r\n[Distillation-v1.1](https://huggingface.co/Tencent-Hunyuan/Distillation-v1.1) for more details.\r\n* Jun 13, 2024: :truck: The training code is released, offering [full-parameter training](#full-parameter-training) and [LoRA training](#lora).\r\n* Jun 06, 2024: :tada: Hunyuan-DiT is now available in ComfyUI. Please check [ComfyUI](#using-comfyui) for more details.\r\n* Jun 06, 2024: 🚀 We introduce Distillation version for Hunyuan-DiT acceleration, which achieves **50%** acceleration on NVIDIA GPUs. Please check [Distillation](https://huggingface.co/Tencent-Hunyuan/Distillation) for more details.\r\n* Jun 05, 2024: 🤗 Hunyuan-DiT is now available in 🤗 Diffusers! Please check the [example](#using--diffusers) below.\r\n* Jun 04, 2024: :globe_with_meridians: Support Tencent Cloud links to download the pretrained models! Please check the [links](#-download-pretrained-models) below.\r\n* May 22, 2024: 🚀 We introduce TensorRT version for Hunyuan-DiT acceleration, which achieves **47%** acceleration on NVIDIA GPUs. Please check [TensorRT-libs](https://huggingface.co/Tencent-Hunyuan/TensorRT-libs) for instructions.\r\n* May 22, 2024: 💬 We support demo running multi-turn text2image generation now. Please check the [script](#using-gradio) below.\r\n\r\n## 🤖 Try it on the web\r\n\r\nWelcome to our web-based [**Tencent Hunyuan Bot**](https://hunyuan.tencent.com/bot/chat), where you can explore our innovative products! Just input the suggested prompts below or any other **imaginative prompts containing drawing-related keywords** to activate the Hunyuan text-to-image generation feature.  Unleash your creativity and create any picture you desire, **all for free!**\r\n\r\nYou can use simple prompts similar to natural language text\r\n\r\n\u003e 画一只穿着西装的猪\r\n\u003e\r\n\u003e draw a pig in a suit\r\n\u003e\r\n\u003e 生成一幅画，赛博朋克风，跑车\r\n\u003e \r\n\u003e generate a painting, cyberpunk style, sports car\r\n\r\nor multi-turn language interactions to create the picture. \r\n\r\n\u003e 画一个木制的鸟\r\n\u003e\r\n\u003e draw a wooden bird\r\n\u003e\r\n\u003e 变成玻璃的\r\n\u003e\r\n\u003e turn into glass\r\n\r\n## 📑 Open-source Plan\r\n\r\n- Hunyuan-DiT (Text-to-Image Model)\r\n  - [x] Inference \r\n  - [x] Checkpoints \r\n  - [x] Distillation Version\r\n  - [x] TensorRT Version\r\n  - [x] Training\r\n  - [x] Lora\r\n  - [x] Controlnet (Pose, Canny, Depth)\r\n  - [x] 6GB GPU VRAM Inference \r\n  - [ ] IP-adapter\r\n  - [ ] Hunyuan-DiT-S checkpoints (0.7B model)\r\n- Mllm\r\n  - Hunyuan-Captioner (Re-caption the raw image-text pairs)\r\n    - [x] Inference\r\n  - [Hunyuan-DialogGen](https://github.com/Centaurusalpha/DialogGen) (Prompt Enhancement Model)\r\n    - [x] Inference\r\n- [X] Web Demo (Gradio) \r\n- [x] Multi-turn T2I Demo (Gradio)\r\n- [X] Cli Demo \r\n- [X] ComfyUI\r\n- [X] Diffusers\r\n- [X] Kohya\r\n- [ ] WebUI\r\n\r\n\r\n## Contents\r\n- [Hunyuan-DiT : A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding](#hunyuan-dit--a-powerful-multi-resolution-diffusion-transformer-with-fine-grained-chinese-understanding)\r\n  - [🔥🔥🔥 News!!](#-news)\r\n  - [🤖 Try it on the web](#-try-it-on-the-web)\r\n  - [📑 Open-source Plan](#-open-source-plan)\r\n  - [Contents](#contents)\r\n  - [**Abstract**](#abstract)\r\n  - [🎉 **Hunyuan-DiT Key Features**](#-hunyuan-dit-key-features)\r\n    - [**Chinese-English Bilingual DiT Architecture**](#chinese-english-bilingual-dit-architecture)\r\n    - [Multi-turn Text2Image Generation](#multi-turn-text2image-generation)\r\n  - [📈 Comparisons](#-comparisons)\r\n  - [🎥 Visualization](#-visualization)\r\n  - [📜 Requirements](#-requirements)\r\n  - [🛠️ Dependencies and Installation](#️-dependencies-and-installation)\r\n    - [Installation Guide for Linux](#installation-guide-for-linux)\r\n  - [🧱 Download Pretrained Models](#-download-pretrained-models)\r\n        - [1. Using HF-Mirror](#1-using-hf-mirror)\r\n        - [2. Resume Download](#2-resume-download)\r\n  - [:truck: Training](#truck-training)\r\n    - [Data Preparation](#data-preparation)\r\n    - [Full-parameter Training](#full-parameter-training)\r\n    - [LoRA](#lora)\r\n  - [🔑 Inference](#-inference)\r\n    - [6GB GPU VRAM Inference](#6gb-gpu-vram-inference)\r\n    - [Using Gradio](#using-gradio)\r\n    - [Using 🤗 Diffusers](#using--diffusers)\r\n    - [Using Command Line](#using-command-line)\r\n    - [More Configurations](#more-configurations)\r\n    - [Using ComfyUI](#using-comfyui)\r\n    - [Using Kohya](#using-kohya)\r\n    - [Using Previous versions](#using-previous-versions)\r\n  - [:building\\_construction: Adapter](#building_construction-adapter)\r\n    - [ControlNet](#controlnet)\r\n  - [:art: Hunyuan-Captioner](#art-hunyuan-captioner)\r\n    - [Examples](#examples)\r\n    - [Instructions](#instructions)\r\n    - [Inference](#inference)\r\n    - [Gradio](#gradio)\r\n  - [🚀 Acceleration (for Linux)](#-acceleration-for-linux)\r\n  - [🔗 BibTeX](#-bibtex)\r\n  - [Start History](#start-history)\r\n\r\n## **Abstract**\r\n\r\nWe present Hunyuan-DiT, a text-to-image diffusion transformer with fine-grained understanding of both English and Chinese. To construct Hunyuan-DiT, we carefully designed the transformer structure, text encoder, and positional encoding. We also build from scratch a whole data pipeline to update and evaluate data for iterative model optimization. For fine-grained language understanding, we train a Multimodal Large Language Model to refine the captions of the images. Finally, Hunyuan-DiT can perform multi-round multi-modal dialogue with users, generating and refining images according to the context.\r\nThrough our carefully designed holistic human evaluation protocol with more than 50 professional human evaluators, Hunyuan-DiT sets a new state-of-the-art in Chinese-to-image generation compared with other open-source models.\r\n\r\n\r\n## 🎉 **Hunyuan-DiT Key Features**\r\n### **Chinese-English Bilingual DiT Architecture**\r\nHunyuan-DiT is a diffusion model in the latent space, as depicted in figure below. Following the Latent Diffusion Model, we use a pre-trained Variational Autoencoder (VAE) to compress the images into low-dimensional latent spaces and train a diffusion model to learn the data distribution with diffusion models. Our diffusion model is parameterized with a transformer. To encode the text prompts, we leverage a combination of pre-trained bilingual (English and Chinese) CLIP and multilingual T5 encoder.\r\n\u003cp align=\"center\"\u003e\r\n  \u003cimg src=\"https://raw.githubusercontent.com/Tencent/HunyuanDiT/main/asset/framework.png\"  height=450\u003e\r\n\u003c/p\u003e\r\n\r\n### Multi-turn Text2Image Generation\r\nUnderstanding natural language instructions and performing multi-turn interaction with users are important for a\r\ntext-to-image system. It can help build a dynamic and iterative creation process that bring the user’s idea into reality\r\nstep by step. In this section, we will detail how we empower Hunyuan-DiT with the ability to perform multi-round\r\nconversations and image generation. We train MLLM to understand the multi-round user dialogue\r\nand output the new text prompt for image generation.\r\n\u003cp align=\"center\"\u003e\r\n  \u003cimg src=\"https://raw.githubusercontent.com/Tencent/HunyuanDiT/main/asset/mllm.png\"  height=300\u003e\r\n\u003c/p\u003e\r\n\r\n## 📈 Comparisons\r\nIn order to comprehensively compare the generation capabilities of HunyuanDiT and other models, we constructed a 4-dimensional test set, including Text-Image Consistency, Excluding AI Artifacts, Subject Clarity, Aesthetic. More than 50 professional evaluators performs the evaluation.\r\n\r\n\u003cp align=\"center\"\u003e\r\n\u003ctable\u003e \r\n\u003cthead\u003e \r\n\u003ctr\u003e \r\n    \u003cth rowspan=\"2\"\u003eModel\u003c/th\u003e \u003cth rowspan=\"2\"\u003eOpen Source\u003c/th\u003e \u003cth\u003eText-Image Consistency (%)\u003c/th\u003e \u003cth\u003eExcluding AI Artifacts (%)\u003c/th\u003e \u003cth\u003eSubject Clarity (%)\u003c/th\u003e \u003cth rowspan=\"2\"\u003eAesthetics (%)\u003c/th\u003e \u003cth rowspan=\"2\"\u003eOverall (%)\u003c/th\u003e \r\n\u003c/tr\u003e \r\n\u003c/thead\u003e \r\n\u003ctbody\u003e \r\n\u003ctr\u003e \r\n    \u003ctd\u003eSDXL\u003c/td\u003e \u003ctd\u003e ✔ \u003c/td\u003e \u003ctd\u003e64.3\u003c/td\u003e \u003ctd\u003e60.6\u003c/td\u003e \u003ctd\u003e91.1\u003c/td\u003e \u003ctd\u003e76.3\u003c/td\u003e \u003ctd\u003e42.7\u003c/td\u003e \r\n\u003c/tr\u003e \r\n\u003ctr\u003e \r\n    \u003ctd\u003ePixArt-α\u003c/td\u003e \u003ctd\u003e ✔\u003c/td\u003e \u003ctd\u003e68.3\u003c/td\u003e \u003ctd\u003e60.9\u003c/td\u003e \u003ctd\u003e93.2\u003c/td\u003e \u003ctd\u003e77.5\u003c/td\u003e \u003ctd\u003e45.5\u003c/td\u003e \r\n\u003c/tr\u003e \r\n\u003ctr\u003e \r\n    \u003ctd\u003ePlayground 2.5\u003c/td\u003e \u003ctd\u003e✔\u003c/td\u003e \u003ctd\u003e71.9\u003c/td\u003e \u003ctd\u003e70.8\u003c/td\u003e \u003ctd\u003e94.9\u003c/td\u003e \u003ctd\u003e83.3\u003c/td\u003e \u003ctd\u003e54.3\u003c/td\u003e \r\n\u003c/tr\u003e \r\n\r\n\u003ctr\u003e \r\n    \u003ctd\u003eSD 3\u003c/td\u003e \u003ctd\u003e\u0026#10008\u003c/td\u003e \u003ctd\u003e77.1\u003c/td\u003e \u003ctd\u003e69.3\u003c/td\u003e \u003ctd\u003e94.6\u003c/td\u003e \u003ctd\u003e82.5\u003c/td\u003e \u003ctd\u003e56.7\u003c/td\u003e \r\n    \r\n\u003c/tr\u003e \r\n\u003ctr\u003e \r\n    \u003ctd\u003eMidJourney v6\u003c/td\u003e\u003ctd\u003e\u0026#10008\u003c/td\u003e \u003ctd\u003e73.5\u003c/td\u003e \u003ctd\u003e80.2\u003c/td\u003e \u003ctd\u003e93.5\u003c/td\u003e \u003ctd\u003e87.2\u003c/td\u003e \u003ctd\u003e63.3\u003c/td\u003e \r\n\u003c/tr\u003e \r\n\u003ctr\u003e \r\n    \u003ctd\u003eDALL-E 3\u003c/td\u003e\u003ctd\u003e\u0026#10008\u003c/td\u003e \u003ctd\u003e83.9\u003c/td\u003e \u003ctd\u003e80.3\u003c/td\u003e \u003ctd\u003e96.5\u003c/td\u003e \u003ctd\u003e89.4\u003c/td\u003e \u003ctd\u003e71.0\u003c/td\u003e \r\n\u003c/tr\u003e \r\n\u003ctr style=\"font-weight: bold; background-color: #f2f2f2;\"\u003e \r\n    \u003ctd\u003eHunyuan-DiT\u003c/td\u003e\u003ctd\u003e✔\u003c/td\u003e \u003ctd\u003e74.2\u003c/td\u003e \u003ctd\u003e74.3\u003c/td\u003e \u003ctd\u003e95.4\u003c/td\u003e \u003ctd\u003e86.6\u003c/td\u003e \u003ctd\u003e59.0\u003c/td\u003e \r\n\u003c/tr\u003e\r\n\u003c/tbody\u003e\r\n\u003c/table\u003e\r\n\u003c/p\u003e\r\n\r\n## 🎥 Visualization\r\n\r\n* **Chinese Elements**\r\n\u003cp align=\"center\"\u003e\r\n  \u003cimg src=\"https://raw.githubusercontent.com/Tencent/HunyuanDiT/main/asset/chinese elements understanding.png\"  height=220\u003e\r\n\u003c/p\u003e\r\n\r\n* **Long Text Input**\r\n\r\n\r\n\u003cp align=\"center\"\u003e\r\n  \u003cimg src=\"https://raw.githubusercontent.com/Tencent/HunyuanDiT/main/asset/long text understanding.png\"  height=310\u003e\r\n\u003c/p\u003e\r\n\r\n* **Multi-turn Text2Image Generation**\r\n\r\nhttps://github.com/Tencent/tencent.github.io/assets/27557933/94b4dcc3-104d-44e1-8bb2-dc55108763d1\r\n\r\n\r\n\r\n---\r\n\r\n## 📜 Requirements\r\n\r\nThis repo consists of DialogGen (a prompt enhancement model) and Hunyuan-DiT (a text-to-image model).\r\n\r\nThe following table shows the requirements for running the models (batch size = 1):\r\n\r\n|          Model          | --load-4bit (DialogGen) | GPU Peak Memory |       GPU       |\r\n|:-----------------------:|:-----------------------:|:---------------:|:---------------:|\r\n| DialogGen + Hunyuan-DiT |            ✘            |       32G       |      A100       |\r\n| DialogGen + Hunyuan-DiT |            ✔            |       22G       |      A100       |\r\n|       Hunyuan-DiT       |            -            |       11G       |      A100       |\r\n|       Hunyuan-DiT       |            -            |       14G       | RTX3090/RTX4090 |\r\n\r\n* An NVIDIA GPU with CUDA support is required. \r\n  * We have tested V100 and A100 GPUs.\r\n  * **Minimum**: The minimum GPU memory required is 11GB.\r\n  * **Recommended**: We recommend using a GPU with 32GB of memory for better generation quality.\r\n* Tested operating system: Linux\r\n\r\n## 🛠️ Dependencies and Installation\r\n\r\nBegin by cloning the repository:\r\n```shell\r\ngit clone https://github.com/tencent/HunyuanDiT\r\ncd HunyuanDiT\r\n```\r\n\r\n### Installation Guide for Linux\r\n\r\nWe provide an `environment.yml` file for setting up a Conda environment.\r\nConda's installation instructions are available [here](https://docs.anaconda.com/free/miniconda/index.html).\r\n\r\nWe recommend CUDA versions 11.7 and 12.0+.\r\n\r\n```shell\r\n# 1. Prepare conda environment\r\nconda env create -f environment.yml\r\n\r\n# 2. Activate the environment\r\nconda activate HunyuanDiT\r\n\r\n# 3. Install pip dependencies\r\npython -m pip install -r requirements.txt\r\n\r\n# 4. Install flash attention v2 for acceleration (requires CUDA 11.6 or above)\r\npython -m pip install git+https://github.com/Dao-AILab/flash-attention.git@v2.1.2.post3\r\n```\r\n\r\nAdditionally, you can also use docker to set up the environment.\r\n```shell\r\n# 1. Use the following link to download the docker image tar file.\r\n# For CUDA 12\r\nwget https://dit.hunyuan.tencent.com/download/HunyuanDiT/hunyuan_dit_cu12.tar\r\n# For CUDA 11\r\nwget https://dit.hunyuan.tencent.com/download/HunyuanDiT/hunyuan_dit_cu11.tar\r\n\r\n# 2. Import the docker tar file and show the image meta information\r\n# For CUDA 12\r\ndocker load -i hunyuan_dit_cu12.tar\r\n# For CUDA 11\r\ndocker load -i hunyuan_dit_cu11.tar  \r\n\r\ndocker image ls\r\n\r\n# 3. Run the container based on the image\r\ndocker run -dit --gpus all --init --net=host --uts=host --ipc=host --name hunyuandit --security-opt=seccomp=unconfined --ulimit=stack=67108864 --ulimit=memlock=-1 --privileged  docker_image_tag\r\n```\r\n\r\n## 🧱 Download Pretrained Models\r\nTo download the model, first install the huggingface-cli. (Detailed instructions are available [here](https://huggingface.co/docs/huggingface_hub/guides/cli).)\r\n\r\n```shell\r\npython -m pip install \"huggingface_hub[cli]\"\r\n```\r\n\r\nThen download the model using the following commands:\r\n\r\n```shell\r\n# Create a directory named 'ckpts' where the model will be saved, fulfilling the prerequisites for running the demo.\r\nmkdir ckpts\r\n# Use the huggingface-cli tool to download the model.\r\n# The download time may vary from 10 minutes to 1 hour depending on network conditions.\r\nhuggingface-cli download Tencent-Hunyuan/HunyuanDiT-v1.2 --local-dir ./ckpts\r\n```\r\n\r\n\u003cdetails\u003e\r\n\u003csummary\u003e💡Tips for using huggingface-cli (network problem)\u003c/summary\u003e\r\n\r\n##### 1. Using HF-Mirror\r\n\r\nIf you encounter slow download speeds in China, you can try a mirror to speed up the download process. For example,\r\n\r\n```shell\r\nHF_ENDPOINT=https://hf-mirror.com huggingface-cli download Tencent-Hunyuan/HunyuanDiT-v1.2 --local-dir ./ckpts\r\n```\r\n\r\n##### 2. Resume Download\r\n\r\n`huggingface-cli` supports resuming downloads. If the download is interrupted, you can just rerun the download \r\ncommand to resume the download process.\r\n\r\nNote: If an `No such file or directory: 'ckpts/.huggingface/.gitignore.lock'` like error occurs during the download \r\nprocess, you can ignore the error and rerun the download command.\r\n\r\n\u003c/details\u003e\r\n\r\n---\r\n\r\nAll models will be automatically downloaded. For more information about the model, visit the Hugging Face repository [here](https://huggingface.co/Tencent-Hunyuan/HunyuanDiT).\r\n\r\n|       Model       | #Params |                                        Huggingface Download URL                                        |                                   Tencent Cloud Download URL                                   |\r\n|:-----------------:|:-------:|:------------------------------------------------------------------------------------------------------:|:----------------------------------------------------------------------------------------------:|\r\n|        mT5        |  1.6B   |               [mT5](https://huggingface.co/Tencent-Hunyuan/HunyuanDiT/tree/main/t2i/mt5)               |               [mT5](https://dit.hunyuan.tencent.com/download/HunyuanDiT/mt5.zip)               |\r\n|       CLIP        |  350M   |       [CLIP](https://huggingface.co/Tencent-Hunyuan/HunyuanDiT/tree/main/t2i/clip_text_encoder)        |       [CLIP](https://dit.hunyuan.tencent.com/download/HunyuanDiT/clip_text_encoder.zip)        |\r\n|     Tokenizer     |  -      |         [Tokenizer](https://huggingface.co/Tencent-Hunyuan/HunyuanDiT/tree/main/t2i/tokenizer)         |         [Tokenizer](https://dit.hunyuan.tencent.com/download/HunyuanDiT/tokenizer.zip)         |\r\n|     DialogGen     |  7.0B   |           [DialogGen](https://huggingface.co/Tencent-Hunyuan/HunyuanDiT/tree/main/dialoggen)           |         [DialogGen](https://dit.hunyuan.tencent.com/download/HunyuanDiT/dialoggen.zip)         |\r\n| sdxl-vae-fp16-fix |   83M   | [sdxl-vae-fp16-fix](https://huggingface.co/Tencent-Hunyuan/HunyuanDiT/tree/main/t2i/sdxl-vae-fp16-fix) | [sdxl-vae-fp16-fix](https://dit.hunyuan.tencent.com/download/HunyuanDiT/sdxl-vae-fp16-fix.zip) |\r\n| Hunyuan-DiT-v1.0  |  1.5B   |          [Hunyuan-DiT](https://huggingface.co/Tencent-Hunyuan/HunyuanDiT/tree/main/t2i/model)          |       [Hunyuan-DiT-v1.0](https://dit.hunyuan.tencent.com/download/HunyuanDiT/model.zip)        |\r\n| Hunyuan-DiT-v1.1  |  1.5B   |     [Hunyuan-DiT-v1.1](https://huggingface.co/Tencent-Hunyuan/HunyuanDiT-v1.1/tree/main/t2i/model)     |     [Hunyuan-DiT-v1.1](https://dit.hunyuan.tencent.com/download/HunyuanDiT/model-v1_1.zip)     |\r\n| Hunyuan-DiT-v1.2  |  1.5B   |     [Hunyuan-DiT-v1.2](https://huggingface.co/Tencent-Hunyuan/HunyuanDiT-v1.2/tree/main/t2i/model)     |     [Hunyuan-DiT-v1.2](https://dit.hunyuan.tencent.com/download/HunyuanDiT/model-v1_2.zip)     |\r\n|     Data demo     |  -      |                                                   -                                                    |         [Data demo](https://dit.hunyuan.tencent.com/download/HunyuanDiT/data_demo.zip)         |\r\n\r\n## :truck: Training\r\n\r\n### Data Preparation\r\n\r\n  Refer to the commands below to prepare the training data. \r\n  \r\n  1. Install dependencies\r\n  \r\n      We offer an efficient data management library, named IndexKits, supporting the management of reading hundreds of millions of data during training, see more in [docs](./IndexKits/README.md).\r\n      ```shell\r\n      # 1 Install dependencies\r\n      cd HunyuanDiT\r\n      pip install -e ./IndexKits\r\n     ```\r\n  2. Data download \r\n  \r\n     Feel free to download the [data demo](https://dit.hunyuan.tencent.com/download/HunyuanDiT/data_demo.zip).\r\n     ```shell\r\n     # 2 Data download\r\n     wget -O ./dataset/data_demo.zip https://dit.hunyuan.tencent.com/download/HunyuanDiT/data_demo.zip\r\n     unzip ./dataset/data_demo.zip -d ./dataset\r\n     mkdir ./dataset/porcelain/arrows ./dataset/porcelain/jsons\r\n     ```\r\n  3. Data conversion \r\n  \r\n     Create a CSV file for training data with the fields listed in the table below.\r\n    \r\n     |    Fields       | Required  |  Description     |   Example   |\r\n     |:---------------:| :------:  |:----------------:|:-----------:|\r\n     |   `image_path`  | Required  |  image path               |     `./dataset/porcelain/images/0.png`        | \r\n     |   `text_zh`     | Required  |    text               |  青花瓷风格，一只蓝色的鸟儿站在蓝色的花瓶上，周围点缀着白色花朵，背景是白色 | \r\n     |   `md5`         | Optional  |    image md5 (Message Digest Algorithm 5)  |    `d41d8cd98f00b204e9800998ecf8427e`         | \r\n     |   `width`       | Optional  |    image width    |     `1024 `       | \r\n     |   `height`      | Optional  |    image height   |    ` 1024 `       | \r\n     \r\n     \u003e ⚠️ Optional fields like MD5, width, and height can be omitted. If omitted, the script below will automatically calculate them. This process can be time-consuming when dealing with large-scale training data.\r\n  \r\n     We utilize [Arrow](https://github.com/apache/arrow) for training data format, offering a standard and efficient in-memory data representation. A conversion script is provided to transform CSV files into Arrow format.\r\n     ```shell  \r\n     # 3 Data conversion \r\n     python ./hydit/data_loader/csv2arrow.py ./dataset/porcelain/csvfile/image_text.csv ./dataset/porcelain/arrows 1\r\n     ```\r\n  \r\n  4. Data Selection and Configuration File Creation \r\n     \r\n      We configure the training data through YAML files. In these files, you can set up standard data processing strategies for filtering, copying, deduplicating, and more regarding the training data. For more details, see [./IndexKits](IndexKits/docs/MakeDataset.md).\r\n  \r\n      For a sample file, please refer to [file](./dataset/yamls/porcelain.yaml). For a full parameter configuration file, see [file](./IndexKits/docs/MakeDataset.md).\r\n  \r\n     \r\n  5. Create training data index file using YAML file.\r\n    \r\n     ```shell\r\n      # Single Resolution Data Preparation\r\n      idk base -c dataset/yamls/porcelain.yaml -t dataset/porcelain/jsons/porcelain.json\r\n   \r\n      # Multi Resolution Data Preparation     \r\n      idk multireso -c dataset/yamls/porcelain_mt.yaml -t dataset/porcelain/jsons/porcelain_mt.json\r\n      ```\r\n   \r\n  The directory structure for `porcelain` dataset is:\r\n\r\n  ```shell\r\n   cd ./dataset\r\n  \r\n   porcelain\r\n      ├──images/  (image files)\r\n      │  ├──0.png\r\n      │  ├──1.png\r\n      │  ├──......\r\n      ├──csvfile/  (csv files containing text-image pairs)\r\n      │  ├──image_text.csv\r\n      ├──arrows/  (arrow files containing all necessary training data)\r\n      │  ├──00000.arrow\r\n      │  ├──00001.arrow\r\n      │  ├──......\r\n      ├──jsons/  (final training data index files which read data from arrow files during training)\r\n      │  ├──porcelain.json\r\n      │  ├──porcelain_mt.json\r\n   ```\r\n\r\n### Full-parameter Training\r\n  \r\n  **Requirement:** \r\n  1. The minimum requriment is a single GPU with at least 20GB memory, but we recommend to use a GPU with about 30 GB memory to avoid host memory offloading. \r\n  2. Additionally, we encourage users to leverage the multiple GPUs across different nodes to speed up training on large datasets. \r\n  \r\n  **Notice:**\r\n  1. Personal users can also use the light-weight Kohya to finetune the model with about 16 GB memory. Currently, we are trying to further reduce the memory usage of our industry-level framework for personal users. \r\n  2. If you have enough GPU memory, please try to remove  `--cpu-offloading` or `--gradient-checkpointing` for less time costs.\r\n\r\n  Specifically for distributed training, you have the flexibility to control **single-node** / **multi-node** training by adjusting parameters such as `--hostfile` and `--master_addr`. For more details, see [link](https://www.deepspeed.ai/getting-started/#resource-configuration-multi-node).\r\n\r\n  ```shell\r\n  # Single Resolution Training\r\n  PYTHONPATH=./ sh hydit/train.sh --index-file dataset/porcelain/jsons/porcelain.json\r\n  \r\n  # Multi Resolution Training\r\n  PYTHONPATH=./ sh hydit/train.sh --index-file dataset/porcelain/jsons/porcelain_mt.json --multireso --reso-step 64\r\n  \r\n  # Training with old version of HunyuanDiT (\u003c= v1.1)\r\n  PYTHONPATH=./ sh hydit/train_v1.1.sh --index-file dataset/porcelain/jsons/porcelain.json\r\n  ```\r\n\r\n  After checkpoints are saved, you can use the following command to evaluate the model.\r\n  ```shell\r\n  # Inference\r\n    #   You should replace the 'log_EXP/xxx/checkpoints/final.pt' with your actual path.\r\n  python sample_t2i.py --infer-mode fa --prompt \"青花瓷风格，一只可爱的哈士奇\" --no-enhance --dit-weight log_EXP/xxx/checkpoints/final.pt --load-key module\r\n  \r\n  # Old version of HunyuanDiT (\u003c= v1.1)\r\n  #   You should replace the 'log_EXP/xxx/checkpoints/final.pt' with your actual path.\r\n  python sample_t2i.py --infer-mode fa --prompt \"青花瓷风格，一只可爱的哈士奇\" --model-root ./HunyuanDiT-v1.1 --use-style-cond --size-cond 1024 1024 --beta-end 0.03 --no-enhance --dit-weight log_EXP/xxx/checkpoints/final.pt --load-key module\r\n  ```\r\n\r\n### LoRA\r\n\r\n\r\n\r\nWe provide training and inference scripts for LoRA, detailed in the [./lora](./lora/README.md). \r\n\r\n  ```shell\r\n  # Training for porcelain LoRA.\r\n  PYTHONPATH=./ sh lora/train_lora.sh --index-file dataset/porcelain/jsons/porcelain.json\r\n\r\n  # Inference using trained LORA weights.\r\n  python sample_t2i.py --infer-mode fa --prompt \"青花瓷风格，一只小狗\"  --no-enhance --lora-ckpt log_EXP/001-lora_porcelain_ema_rank64/checkpoints/0001000.pt\r\n  ```\r\n We offer two types of trained LoRA weights for `porcelain` and `jade`, see details at [links](https://huggingface.co/Tencent-Hunyuan/HYDiT-LoRA)\r\n  ```shell\r\n  cd HunyuanDiT\r\n  # Use the huggingface-cli tool to download the model.\r\n  huggingface-cli download Tencent-Hunyuan/HYDiT-LoRA --local-dir ./ckpts/t2i/lora\r\n  \r\n  # Quick start\r\n  python sample_t2i.py --infer-mode fa --prompt \"青花瓷风格，一只猫在追蝴蝶\"  --no-enhance --load-key ema --lora-ckpt ./ckpts/t2i/lora/porcelain\r\n  ```\r\n \u003ctable\u003e\r\n  \u003ctr\u003e\r\n    \u003ctd colspan=\"4\" align=\"center\"\u003eExamples of training data\u003c/td\u003e\r\n  \u003c/tr\u003e\r\n  \r\n  \u003ctr\u003e\r\n    \u003ctd align=\"center\"\u003e\u003cimg src=\"lora/asset/porcelain/train/0.png\" alt=\"Image 0\" width=\"200\"/\u003e\u003c/td\u003e\r\n    \u003ctd align=\"center\"\u003e\u003cimg src=\"lora/asset/porcelain/train/1.png\" alt=\"Image 1\" width=\"200\"/\u003e\u003c/td\u003e\r\n    \u003ctd align=\"center\"\u003e\u003cimg src=\"lora/asset/porcelain/train/2.png\" alt=\"Image 2\" width=\"200\"/\u003e\u003c/td\u003e\r\n    \u003ctd align=\"center\"\u003e\u003cimg src=\"lora/asset/porcelain/train/3.png\" alt=\"Image 3\" width=\"200\"/\u003e\u003c/td\u003e\r\n  \u003c/tr\u003e\r\n  \u003ctr\u003e\r\n    \u003ctd align=\"center\"\u003e青花瓷风格，一只蓝色的鸟儿站在蓝色的花瓶上，周围点缀着白色花朵，背景是白色 （Porcelain style, a blue bird stands on a blue vase, surrounded by white flowers, with a white background.\r\n）\u003c/td\u003e\r\n    \u003ctd align=\"center\"\u003e青花瓷风格，这是一幅蓝白相间的陶瓷盘子，上面描绘着一只狐狸和它的幼崽在森林中漫步，背景是白色 （Porcelain style, this is a blue and white ceramic plate depicting a fox and its cubs strolling in the forest, with a white background.）\u003c/td\u003e\r\n    \u003ctd align=\"center\"\u003e青花瓷风格，在黑色背景上，一只蓝色的狼站在蓝白相间的盘子上，周围是树木和月亮 （Porcelain style, on a black background, a blue wolf stands on a blue and white plate, surrounded by trees and the moon.）\u003c/td\u003e\r\n    \u003ctd align=\"center\"\u003e青花瓷风格，在蓝色背景上，一只蓝色蝴蝶和白色花朵被放置在中央 （Porcelain style, on a blue background, a blue butterfly and white flowers are placed in the center.）\u003c/td\u003e\r\n  \u003c/tr\u003e\r\n  \u003ctr\u003e\r\n    \u003ctd colspan=\"4\" align=\"center\"\u003eExamples of inference results\u003c/td\u003e\r\n  \u003c/tr\u003e\r\n  \u003ctr\u003e\r\n    \u003ctd align=\"center\"\u003e\u003cimg src=\"lora/asset/porcelain/inference/0.png\" alt=\"Image 4\" width=\"200\"/\u003e\u003c/td\u003e\r\n    \u003ctd align=\"center\"\u003e\u003cimg src=\"lora/asset/porcelain/inference/1.png\" alt=\"Image 5\" width=\"200\"/\u003e\u003c/td\u003e\r\n    \u003ctd align=\"center\"\u003e\u003cimg src=\"lora/asset/porcelain/inference/2.png\" alt=\"Image 6\" width=\"200\"/\u003e\u003c/td\u003e\r\n    \u003ctd align=\"center\"\u003e\u003cimg src=\"lora/asset/porcelain/inference/3.png\" alt=\"Image 7\" width=\"200\"/\u003e\u003c/td\u003e\r\n  \u003c/tr\u003e\r\n  \u003ctr\u003e\r\n    \u003ctd align=\"center\"\u003e青花瓷风格，苏州园林 （Porcelain style,  Suzhou Gardens.）\u003c/td\u003e\r\n    \u003ctd align=\"center\"\u003e青花瓷风格，一朵荷花 （Porcelain style,  a lotus flower.）\u003c/td\u003e\r\n    \u003ctd align=\"center\"\u003e青花瓷风格，一只羊（Porcelain style, a sheep.）\u003c/td\u003e\r\n    \u003ctd align=\"center\"\u003e青花瓷风格，一个女孩在雨中跳舞（Porcelain style, a girl dancing in the rain.）\u003c/td\u003e\r\n  \u003c/tr\u003e\r\n  \r\n\u003c/table\u003e\r\n\r\n\r\n## 🔑 Inference\r\n\r\n### 6GB GPU VRAM Inference\r\nRunning HunyuanDiT in under 6GB GPU VRAM is available now based on [diffusers](https://huggingface.co/docs/diffusers/main/en/api/pipelines/hunyuandit). Here we provide instructions and demo for your quick start.\r\n\r\n\u003e The 6GB version supports Nvidia Ampere architecture series graphics cards such as RTX 3070/3080/4080/4090, A100, and so on.\r\n\r\nThe only thing you need do is to install the following library:\r\n\r\n```bash\r\npip install -U bitsandbytes\r\npip install git+https://github.com/huggingface/diffusers\r\npip install torch==2.0.0\r\n```\r\n\r\nThen you can enjoy your HunyuanDiT text-to-image journey under 6GB GPU VRAM directly!\r\n\r\nHere is a demo for you.\r\n\r\n```bash\r\ncd HunyuanDiT\r\n\r\n# Quick start\r\nmodel_id=Tencent-Hunyuan/HunyuanDiT-v1.2-Diffusers-Distilled\r\nprompt=一个宇航员在骑马\r\ninfer_steps=50\r\nguidance_scale=6\r\npython3 lite/inference.py ${model_id} ${prompt} ${infer_steps} ${guidance_scale}\r\n```\r\n\r\nMore details can be found in [./lite](lite/README.md).\r\n\r\n\r\n### Using Gradio\r\n\r\nMake sure the conda environment is activated before running the following command.\r\n\r\n```shell\r\n# By default, we start a Chinese UI. Using Flash Attention for acceleration.\r\npython app/hydit_app.py --infer-mode fa\r\n\r\n# You can disable the enhancement model if the GPU memory is insufficient.\r\n# The enhancement will be unavailable until you restart the app without the `--no-enhance` flag. \r\npython app/hydit_app.py --no-enhance --infer-mode fa\r\n\r\n# Start with English UI\r\npython app/hydit_app.py --lang en --infer-mode fa\r\n\r\n# Start a multi-turn T2I generation UI. \r\n# If your GPU memory is less than 32GB, use '--load-4bit' to enable 4-bit quantization, which requires at least 22GB of memory.\r\npython app/multiTurnT2I_app.py --infer-mode fa\r\n```\r\nThen the demo can be accessed through http://0.0.0.0:443. It should be noted that the 0.0.0.0 here needs to be X.X.X.X with your server IP.\r\n\r\n### Using 🤗 Diffusers\r\n\r\nPlease install PyTorch version 2.0 or higher in advance to satisfy the requirements of the specified version of the diffusers library.  \r\n\r\nInstall 🤗 diffusers, ensuring that the version is at least 0.28.1:\r\n\r\n```shell\r\npip install git+https://github.com/huggingface/diffusers.git\r\n```\r\nor\r\n```shell\r\npip install diffusers\r\n```\r\n\r\nYou can generate images with both Chinese and English prompts using the following Python script:\r\n```py\r\nimport torch\r\nfrom diffusers import HunyuanDiTPipeline\r\n\r\npipe = HunyuanDiTPipeline.from_pretrained(\"Tencent-Hunyuan/HunyuanDiT-v1.2-Diffusers\", torch_dtype=torch.float16)\r\npipe.to(\"cuda\")\r\n\r\n# You may also use English prompt as HunyuanDiT supports both English and Chinese\r\n# prompt = \"An astronaut riding a horse\"\r\nprompt = \"一个宇航员在骑马\"\r\nimage = pipe(prompt).images[0]\r\n```\r\nYou can use our distilled model to generate images even faster:\r\n\r\n```py\r\nimport torch\r\nfrom diffusers import HunyuanDiTPipeline\r\n\r\npipe = HunyuanDiTPipeline.from_pretrained(\"Tencent-Hunyuan/HunyuanDiT-v1.2-Diffusers-Distilled\", torch_dtype=torch.float16)\r\npipe.to(\"cuda\")\r\n\r\n# You may also use English prompt as HunyuanDiT supports both English and Chinese\r\n# prompt = \"An astronaut riding a horse\"\r\nprompt = \"一个宇航员在骑马\"\r\nimage = pipe(prompt, num_inference_steps=25).images[0]\r\n```\r\nMore details can be found in [HunyuanDiT-v1.2-Diffusers-Distilled](https://huggingface.co/Tencent-Hunyuan/HunyuanDiT-v1.2-Diffusers-Distilled)\r\n\r\n**More functions:** For other functions like LoRA and ControlNet, please have a look at the README of [./diffusers](diffusers).\r\n\r\n### Using Command Line\r\n\r\nWe provide several commands to quick start: \r\n\r\n```shell\r\n# Only Text-to-Image. Flash Attention mode\r\npython sample_t2i.py --infer-mode fa --prompt \"渔舟唱晚\" --no-enhance\r\n\r\n# Generate an image with other image sizes.\r\npython sample_t2i.py --infer-mode fa --prompt \"渔舟唱晚\" --image-size 1280 768\r\n\r\n# Prompt Enhancement + Text-to-Image. DialogGen loads with 4-bit quantization, but it may loss performance.\r\npython sample_t2i.py --infer-mode fa --prompt \"渔舟唱晚\"  --load-4bit\r\n\r\n```\r\n\r\nMore example prompts can be found in [example_prompts.txt](example_prompts.txt)\r\n\r\n### More Configurations\r\n\r\nWe list some more useful configurations for easy usage:\r\n\r\n|    Argument     |  Default  |                     Description                     |\r\n|:---------------:|:---------:|:---------------------------------------------------:|\r\n|   `--prompt`    |   None    |        The text prompt for image generation         |\r\n| `--image-size`  | 1024 1024 |           The size of the generated image           |\r\n|    `--seed`     |    42     |        The random seed for generating images        |\r\n| `--infer-steps` |    100    |          The number of steps for sampling           |\r\n|  `--negative`   |     -     |      The negative prompt for image generation       |\r\n| `--infer-mode`  |   torch   |       The inference mode (torch, fa, or trt)        |\r\n|   `--sampler`   |   ddpm    |    The diffusion sampler (ddpm, ddim, or dpmms)     |\r\n| `--no-enhance`  |   False   |        Disable the prompt enhancement model         |\r\n| `--model-root`  |   ckpts   |     The root directory of the model checkpoints     |\r\n|  `--load-key`   |    ema    | Load the student model or EMA model (ema or module) |\r\n|  `--load-4bit`  |   Fasle   |     Load DialogGen model with 4bit quantization     |\r\n\r\n### Using ComfyUI\r\n\r\n- Support two workflows: Standard ComfyUI and Diffusers Wrapper, with the former being recommended.\r\n- Support HunyuanDiT-v1.1 and v1.2.\r\n- Support module, lora and clip lora models trained by Kohya.\r\n- Support module, lora models trained by HunyunDiT official training scripts.\r\n- ControlNet is coming soon.\r\n\r\n![Workflow](comfyui-hydit/img/workflow_v1.2_lora.png)\r\nMore details can be found in [./comfyui-hydit](comfyui-hydit/README.md)\r\n\r\n### Using Kohya\r\n\r\nWe support custom codes for kohya_ss GUI, and sd-scripts training codes for HunyuanDiT.\r\n![dreambooth](kohya_ss-hydit/img/dreambooth.png)\r\nMore details can be found in [./kohya_ss-hydit](kohya_ss-hydit/README.md)\r\n\r\n### Using Previous versions\r\n\r\n* **Hunyuan-DiT \u003c= v1.1**\r\n\r\n```shell\r\n# ============================== v1.1 ==============================\r\n# Download the model\r\nhuggingface-cli download Tencent-Hunyuan/HunyuanDiT-v1.1 --local-dir ./HunyuanDiT-v1.1\r\n# Inference with the model\r\npython sample_t2i.py --infer-mode fa --prompt \"渔舟唱晚\" --model-root ./HunyuanDiT-v1.1 --use-style-cond --size-cond 1024 1024 --beta-end 0.03\r\n\r\n# ============================== v1.0 ==============================\r\n# Download the model\r\nhuggingface-cli download Tencent-Hunyuan/HunyuanDiT --local-dir ./HunyuanDiT-v1.0\r\n# Inference with the model\r\npython sample_t2i.py --infer-mode fa --prompt \"渔舟唱晚\" --model-root ./HunyuanDiT-v1.0 --use-style-cond --size-cond 1024 1024 --beta-end 0.03\r\n```\r\n\r\n## :building_construction: Adapter\r\n\r\n### ControlNet\r\n\r\nWe provide training scripts for ControlNet, detailed in the [./controlnet](./controlnet/README.md). \r\n\r\n  ```shell\r\n  # Training for canny ControlNet.\r\n  PYTHONPATH=./ sh hydit/train_controlnet.sh\r\n  ```\r\n We offer three types of trained ControlNet weights for `canny` `depth` and `pose`, see details at [links](https://huggingface.co/Tencent-Hunyuan/HYDiT-ControlNet)\r\n  ```shell\r\n  cd HunyuanDiT\r\n  # Use the huggingface-cli tool to download the model.\r\n  # We recommend using distilled weights as the base model for ControlNet inference, as our provided pretrained weights are trained on them.\r\n  huggingface-cli download Tencent-Hunyuan/HYDiT-ControlNet-v1.2 --local-dir ./ckpts/t2i/controlnet\r\n  huggingface-cli download Tencent-Hunyuan/Distillation-v1.2 ./pytorch_model_distill.pt --local-dir ./ckpts/t2i/model\r\n  \r\n  # Quick start\r\n  python3 sample_controlnet.py --infer-mode fa --no-enhance --load-key distill --infer-steps 50 --control-type canny --prompt \"在夜晚的酒店门前，一座古老的中国风格的狮子雕像矗立着，它的眼睛闪烁着光芒，仿佛在守护着这座建筑。背景是夜晚的酒店前，构图方式是特写，平视，居中构图。这张照片呈现了真实摄影风格，蕴含了中国雕塑文化，同时展现了神秘氛围\" --condition-image-path controlnet/asset/input/canny.jpg --control-weight 1.0\r\n  \r\n  ```\r\n \r\n \u003ctable\u003e\r\n  \u003ctr\u003e\r\n    \u003ctd colspan=\"3\" align=\"center\"\u003eCondition Input\u003c/td\u003e\r\n  \u003c/tr\u003e\r\n  \r\n   \u003ctr\u003e\r\n    \u003ctd align=\"center\"\u003eCanny ControlNet \u003c/td\u003e\r\n    \u003ctd align=\"center\"\u003eDepth ControlNet \u003c/td\u003e\r\n    \u003ctd align=\"center\"\u003ePose ControlNet \u003c/td\u003e\r\n  \u003c/tr\u003e\r\n\r\n  \u003ctr\u003e\r\n    \u003ctd align=\"center\"\u003e在夜晚的酒店门前，一座古老的中国风格的狮子雕像矗立着，它的眼睛闪烁着光芒，仿佛在守护着这座建筑。背景是夜晚的酒店前，构图方式是特写，平视，居中构图。这张照片呈现了真实摄影风格，蕴含了中国雕塑文化，同时展现了神秘氛围\u003cbr\u003e（At night, an ancient Chinese-style lion statue stands in front of the hotel, its eyes gleaming as if guarding the building. The background is the hotel entrance at night, with a close-up, eye-level, and centered composition. This photo presents a realistic photographic style, embodies Chinese sculpture culture, and reveals a mysterious atmosphere.） \u003c/td\u003e\r\n    \u003ctd align=\"center\"\u003e在茂密的森林中，一只黑白相间的熊猫静静地坐在绿树红花中，周围是山川和海洋。背景是白天的森林，光线充足。照片采用特写、平视和居中构图的方式，呈现出写实的效果\u003cbr\u003e（In the dense forest, a black and white panda sits quietly among the green trees and red flowers, surrounded by mountains and oceans. The background is a daytime forest with ample light. The photo uses a close-up, eye-level, and centered composition to create a realistic effect.） \u003c/td\u003e\r\n    \u003ctd align=\"center\"\u003e在白天的森林中，一位穿着绿色上衣的亚洲女性站在大象旁边。照片采用了中景、平视和居中构图的方式，呈现出写实的效果。这张照片蕴含了人物摄影文化，并展现了宁静的氛围\u003cbr\u003e（In the daytime forest, an Asian woman wearing a green shirt stands beside an elephant. The photo uses a medium shot, eye-level, and centered composition to create a realistic effect. This picture embodies the character photography culture and conveys a serene atmosphere.） \u003c/td\u003e\r\n  \u003c/tr\u003e\r\n\r\n  \u003ctr\u003e\r\n    \u003ctd align=\"center\"\u003e\u003cimg src=\"controlnet/asset/input/canny.jpg\" alt=\"Image 0\" width=\"200\"/\u003e\u003c/td\u003e\r\n    \u003ctd align=\"center\"\u003e\u003cimg src=\"controlnet/asset/input/depth.jpg\" alt=\"Image 1\" width=\"200\"/\u003e\u003c/td\u003e\r\n    \u003ctd align=\"center\"\u003e\u003cimg src=\"controlnet/asset/input/pose.jpg\" alt=\"Image 2\" width=\"200\"/\u003e\u003c/td\u003e\r\n    \r\n  \u003c/tr\u003e\r\n  \r\n  \u003ctr\u003e\r\n    \u003ctd colspan=\"3\" align=\"center\"\u003eControlNet Output\u003c/td\u003e\r\n  \u003c/tr\u003e\r\n\r\n  \u003ctr\u003e\r\n    \u003ctd align=\"center\"\u003e\u003cimg src=\"controlnet/asset/output/canny.jpg\" alt=\"Image 3\" width=\"200\"/\u003e\u003c/td\u003e\r\n    \u003ctd align=\"center\"\u003e\u003cimg src=\"controlnet/asset/output/depth.jpg\" alt=\"Image 4\" width=\"200\"/\u003e\u003c/td\u003e\r\n    \u003ctd align=\"center\"\u003e\u003cimg src=\"controlnet/asset/output/pose.jpg\" alt=\"Image 5\" width=\"200\"/\u003e\u003c/td\u003e\r\n  \u003c/tr\u003e\r\n \r\n\u003c/table\u003e\r\n\r\n## :art: Hunyuan-Captioner\r\nHunyuan-Captioner meets the need of text-to-image techniques by maintaining a high degree of image-text consistency. It can generate high-quality image descriptions from a variety of angles, including object description, objects relationships, background information, image style, etc. Our code is based on [LLaVA](https://github.com/haotian-liu/LLaVA) implementation.\r\n\r\n### Examples\r\n\r\n\u003ctd align=\"center\"\u003e\u003cimg src=\"./asset/caption_demo.jpg\" alt=\"Image 3\" width=\"1200\"/\u003e\u003c/td\u003e\r\n\r\n### Instructions\r\na. Install dependencies\r\n     \r\nThe dependencies and installation are basically the same as the [**base model**](https://huggingface.co/Tencent-Hunyuan/HunyuanDiT-v1.2).\r\n\r\nb. Model download\r\n```shell\r\n# Use the huggingface-cli tool to download the model.\r\nhuggingface-cli download Tencent-Hunyuan/HunyuanCaptioner --local-dir ./ckpts/captioner\r\n```\r\n\r\n### Inference\r\n\r\nOur model supports three different modes including: **directly generating Chinese caption**, **generating Chinese caption based on specific knowledge**, and **directly generating English caption**. The injected information can be either accurate cues or noisy labels (e.g., raw descriptions crawled from the internet). The model is capable of generating reliable and accurate descriptions based on both the inserted information and the image content.\r\n\r\n|Mode           | Prompt Template                           |Description                           | \r\n| ---           | ---                                       | ---                                  |\r\n|caption_zh     | 描述这张图片                               |Caption in Chinese                    | \r\n|insert_content | 根据提示词“{}”,描述这张图片                 |Caption with inserted knowledge| \r\n|caption_en     | Please describe the content of this image |Caption in English                    |\r\n|               |                                           |                                      |\r\n \r\n\r\na. Single picture inference in Chinese\r\n\r\n```bash\r\npython mllm/caption_demo.py --mode \"caption_zh\" --image_file \"mllm/images/demo1.png\" --model_path \"./ckpts/captioner\"\r\n```\r\n\r\nb. Insert specific knowledge into caption\r\n\r\n```bash\r\npython mllm/caption_demo.py --mode \"insert_content\" --content \"宫保鸡丁\" --image_file \"mllm/images/demo2.png\" --model_path \"./ckpts/captioner\"\r\n```\r\n\r\nc. Single picture inference in English\r\n\r\n```bash\r\npython mllm/caption_demo.py --mode \"caption_en\" --image_file \"mllm/images/demo3.png\" --model_path \"./ckpts/captioner\"\r\n```\r\n\r\nd. Multiple pictures inference in Chinese\r\n\r\n```bash\r\n### Convert multiple pictures to csv file. \r\npython mllm/make_csv.py --img_dir \"mllm/images\" --input_file \"mllm/images/demo.csv\"\r\n\r\n### Multiple pictures inference\r\npython mllm/caption_demo.py --mode \"caption_zh\" --input_file \"mllm/images/demo.csv\" --output_file \"mllm/images/demo_res.csv\" --model_path \"./ckpts/captioner\"\r\n```\r\n\r\n(Optional) To convert the output csv file to Arrow format, please refer to [Data Preparation #3](#data-preparation) for detailed instructions. \r\n\r\n\r\n### Gradio \r\nTo launch a Gradio demo locally, please run the following commands one by one. For more detailed instructions, please refer to [LLaVA](https://github.com/haotian-liu/LLaVA). \r\n```bash\r\ncd mllm\r\npython -m llava.serve.controller --host 0.0.0.0 --port 10000\r\n\r\npython -m llava.serve.gradio_web_server --controller http://0.0.0.0:10000 --model-list-mode reload --port 443\r\n\r\npython -m llava.serve.model_worker --host 0.0.0.0 --controller http://0.0.0.0:10000 --port 40000 --worker http://0.0.0.0:40000 --model-path \"../ckpts/captioner\" --model-name LlavaMistral\r\n```\r\nThen the demo can be accessed through http://0.0.0.0:443. It should be noted that the 0.0.0.0 here needs to be X.X.X.X with your server IP.\r\n\r\n## 🚀 Acceleration (for Linux)\r\n\r\n- We provide TensorRT version of HunyuanDiT for inference acceleration (faster than flash attention).\r\nSee [Tencent-Hunyuan/TensorRT-libs](https://huggingface.co/Tencent-Hunyuan/TensorRT-libs) for more details.\r\n\r\n- We provide Distillation version of HunyuanDiT for inference acceleration.\r\nSee [Tencent-Hunyuan/Distillation](https://huggingface.co/Tencent-Hunyuan/Distillation) for more details.\r\n\r\n## 🔗 BibTeX\r\nIf you find [Hunyuan-DiT](https://arxiv.org/abs/2405.08748) or [DialogGen](https://arxiv.org/abs/2403.08857) useful for your research and applications, please cite using this BibTeX:\r\n\r\n```BibTeX\r\n@misc{li2024hunyuandit,\r\n      title={Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding}, \r\n      author={Zhimin Li and Jianwei Zhang and Qin Lin and Jiangfeng Xiong and Yanxin Long and Xinchi Deng and Yingfang Zhang and Xingchao Liu and Minbin Huang and Zedong Xiao and Dayou Chen and Jiajun He and Jiahao Li and Wenyue Li and Chen Zhang and Rongwei Quan and Jianxiang Lu and Jiabin Huang and Xiaoyan Yuan and Xiaoxiao Zheng and Yixuan Li and Jihong Zhang and Chao Zhang and Meng Chen and Jie Liu and Zheng Fang and Weiyan Wang and Jinbao Xue and Yangyu Tao and Jianchen Zhu and Kai Liu and Sihuan Lin and Yifu Sun and Yun Li and Dongdong Wang and Mingtao Chen and Zhichao Hu and Xiao Xiao and Yan Chen and Yuhong Liu and Wei Liu and Di Wang and Yong Yang and Jie Jiang and Qinglin Lu},\r\n      year={2024},\r\n      eprint={2405.08748},\r\n      archivePrefix={arXiv},\r\n      primaryClass={cs.CV}\r\n}\r\n\r\n@article{huang2024dialoggen,\r\n  title={DialogGen: Multi-modal Interactive Dialogue System for Multi-turn Text-to-Image Generation},\r\n  author={Huang, Minbin and Long, Yanxin and Deng, Xinchi and Chu, Ruihang and Xiong, Jiangfeng and Liang, Xiaodan and Cheng, Hong and Lu, Qinglin and Liu, Wei},\r\n  journal={arXiv preprint arXiv:2403.08857},\r\n  year={2024}\r\n}\r\n```\r\n\r\n## Start History\r\n\r\n\u003ca href=\"https://star-history.com/#Tencent/HunyuanDiT\u0026Date\"\u003e\r\n \u003cpicture\u003e\r\n   \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"https://api.star-history.com/svg?repos=Tencent/HunyuanDiT\u0026type=Date\u0026theme=dark\" /\u003e\r\n   \u003csource media=\"(prefers-color-scheme: light)\" srcset=\"https://api.star-history.com/svg?repos=Tencent/HunyuanDiT\u0026type=Date\" /\u003e\r\n   \u003cimg alt=\"Star History Chart\" src=\"https://api.star-history.com/svg?repos=Tencent/HunyuanDiT\u0026type=Date\" /\u003e\r\n \u003c/picture\u003e\r\n\u003c/a\u003e\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftencent%2Fhunyuandit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftencent%2Fhunyuandit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftencent%2Fhunyuandit/lists"}