{"id":13491223,"url":"https://github.com/PixArt-alpha/PixArt-alpha","last_synced_at":"2025-03-28T08:32:48.464Z","repository":{"id":200894578,"uuid":"704092425","full_name":"PixArt-alpha/PixArt-alpha","owner":"PixArt-alpha","description":"PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis","archived":false,"fork":false,"pushed_at":"2024-08-01T06:27:17.000Z","size":62350,"stargazers_count":2765,"open_issues_count":4,"forks_count":177,"subscribers_count":47,"default_branch":"master","last_synced_at":"2024-10-29T15:34:27.347Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://pixart-alpha.github.io/","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PixArt-alpha.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-12T14:16:33.000Z","updated_at":"2024-10-29T13:05:46.000Z","dependencies_parsed_at":"2024-11-19T22:21:38.545Z","dependency_job_id":null,"html_url":"https://github.com/PixArt-alpha/PixArt-alpha","commit_stats":{"total_commits":142,"total_committers":21,"mean_commits":6.761904761904762,"dds":0.6126760563380282,"last_synced_commit":"3e9d36f2f6ba18bb0b122667e1b1c2031a8e06fb"},"previous_names":["pixart-alpha/pixart"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PixArt-alpha%2FPixArt-alpha","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PixArt-alpha%2FPixArt-alpha/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PixArt-alpha%2FPixArt-alpha/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PixArt-alpha%2FPixArt-alpha/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PixArt-alpha","download_url":"https://codeload.github.com/PixArt-alpha/PixArt-alpha/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245996338,"owners_count":20707221,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T19:00:54.694Z","updated_at":"2025-03-28T08:32:48.425Z","avatar_url":"https://github.com/PixArt-alpha.png","language":"Python","funding_links":[],"categories":["Python","其他_机器视觉"],"sub_categories":["网络服务_其他"],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"asset/logo.png\"  height=120\u003e\n\u003c/p\u003e\n\n\n### \u003cdiv align=\"center\"\u003e👉 PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis\u003cdiv\u003e \n### \u003cdiv align=\"center\"\u003e ICLR 2024 Spotlight \u003cdiv\u003e \n\n\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"https://github.com/PixArt-alpha/PixArt-sigma/\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=PixArt-Sigma Code\u0026message=Github\u0026color=blue\u0026logo=github-pages\"\u003e\u003c/a\u003e \u0026ensp;\n\n  \u003ca href=\"https://pixart-alpha.github.io/\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=Project%20Page\u0026message=Github\u0026color=blue\u0026logo=github-pages\"\u003e\u003c/a\u003e \u0026ensp;\n  \u003ca href=\"https://huggingface.co/datasets/PixArt-alpha/SAM-LLaVA-Captions10M\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=SAM-LLaVA\u0026message=HF\u0026color=yellow\"\u003e\u003c/a\u003e \u0026ensp;\n  \u003ca href=\"https://arxiv.org/abs/2310.00426\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=Paper\u0026message=Arxiv:Alpha\u0026color=red\u0026logo=arxiv\"\u003e\u003c/a\u003e \u0026ensp;\n  \u003ca href=\"https://arxiv.org/abs/2401.05252\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=Paper\u0026message=Arxiv:Delta\u0026color=red\u0026logo=arxiv\"\u003e\u003c/a\u003e \u0026ensp;\n  \u003ca href=\"https://discord.gg/rde6eaE5Ta\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=Discuss\u0026message=Discord\u0026color=purple\u0026logo=discord\"\u003e\u003c/a\u003e \u0026ensp;\n  \u003ca href=\"https://huggingface.co/docs/diffusers/main/en/api/pipelines/pixart\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=Usage\u0026message=Diffusers\u0026color=green\u0026\"\u003e\u003c/a\u003e \u0026ensp;\n  \u003ca href=\"https://github.com/city96/ComfyUI_ExtraModels\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=App\u0026message=ComfyUI\u0026\u0026color=green\"\u003e\u003c/a\u003e \u0026ensp;\n\n  \u003ca href=\"https://huggingface.co/spaces/PixArt-alpha/PixArt-alpha\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=Demo PixArt\u0026message=HuggingFace\u0026color=yellow\"\u003e\u003c/a\u003e \u0026ensp;\n  \u003ca href=\"https://huggingface.co/spaces/PixArt-alpha/PixArt-LCM\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=Demo PixArt-LCM\u0026message=HuggingFace\u0026color=yellow\"\u003e\u003c/a\u003e \u0026ensp;\n  \u003ca href=\"https://openxlab.org.cn/apps/detail/PixArt-alpha/PixArt-alpha\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=Demo PixArt\u0026message=OpenXLab\u0026color=purple\"\u003e\u003c/a\u003e \u0026ensp;\n  \u003ca href=\"https://openxlab.org.cn/apps/detail/houshaowei/PixArt-LCM\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=Demo PixArt-LCM\u0026message=OpenXLab\u0026color=purple\"\u003e\u003c/a\u003e \u0026ensp;\n  \u003ca href=\"https://colab.research.google.com/drive/1jZ5UZXk7tcpTfVwnX33dDuefNMcnW9ME?usp=sharing\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=Free%20Trial\u0026message=Google%20Colab\u0026logo=google\u0026color=orange\"\u003e\u003c/a\u003e \u0026ensp;\n\u003c/div\u003e\n\n---\n\nThis repo contains PyTorch model definitions, pre-trained weights and inference/sampling code for our paper exploring \nFast training diffusion models with transformers. You can find more visualizations on our [project page](https://pixart-alpha.github.io/).\n\n\u003cimg src=\"asset/logo.png\" width=\"10%\" alt=\"\" /\u003e **PixArt-α Community**: Join our PixArt-α discord channels \u003ca href=\"https://discord.gg/rde6eaE5Ta\" style=\"text-decoration:none;\"\u003e\n\u003cimg src=\"https://user-images.githubusercontent.com/25839884/218347213-c080267f-cbb6-443e-8532-8e1ed9a58ea9.png\" width=\"3%\" alt=\"\" /\u003e\u003c/a\u003e for discussions. Coders are welcome to contribute.\n\n\u003e [**PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis**](https://pixart-alpha.github.io/)\u003cbr\u003e\n\u003e [Junsong Chen*](https://lawrence-cj.github.io/), [Jincheng Yu*](https://lovesykun.cn/about.html), \n\u003e [Chongjian Ge*](https://chongjiange.github.io/), [Lewei Yao*](https://scholar.google.com/citations?user=hqDyTg8AAAAJ\u0026hl=zh-CN\u0026oi=ao),\n\u003e [Enze Xie](https://xieenze.github.io/)\u0026#8224;,\n\u003e [Yue Wu](https://yuewuhkust.github.io/), [Zhongdao Wang](https://zhongdao.github.io/), \n\u003e [James Kwok](https://www.cse.ust.hk/~jamesk/), [Ping Luo](http://luoping.me/), \n\u003e [Huchuan Lu](https://scholar.google.com/citations?hl=en\u0026user=D3nE0agAAAAJ), \n\u003e [Zhenguo Li](https://scholar.google.com/citations?user=XboZC1AAAAAJ)\n\u003e \u003cbr\u003eHuawei Noah’s Ark Lab, Dalian University of Technology, HKU, HKUST\u003cbr\u003e\n\n\u003e [**PIXART-δ: Fast and Controllable Image Generation with Latent Consistency Models**](https://pixart-alpha.github.io/)\u003cbr\u003e\n\u003e [Junsong Chen](https://lawrence-cj.github.io/), [Yue Wu](https://yuewuhkust.github.io/), [Simian Luo](https://luosiallen.github.io/),  [Enze Xie](https://xieenze.github.io/)\u0026#8224;,\n\u003e [Sayak Paul](https://sayak.dev/), [Ping Luo](http://luoping.me/), [Hang Zhao](), [Zhenguo Li](https://scholar.google.com/citations?user=XboZC1AAAAAJ)\n\u003e \u003cbr\u003eHuawei Noah’s Ark Lab, DLUT, Tsinghua University, HKU, Hugging Face\u003cbr\u003e\n\n---\n## Breaking News 🔥🔥!!\n- (🔥 New) Apr. 12, 2024. 💥 A better version of [PixArt-Σ](https://github.com/PixArt-alpha/PixArt-sigma) training \u0026 inference code, checkpoints are all released!!!\nWelcome to collaborate and contribute. Star 🌟us if you think it is helpful!!\n\n\n- (🔥 New) Jan. 19, 2024. 💥 [PixArt-δ](https://arxiv.org/abs/2401.05252) ControlNet [app_controlnet.py](app/app_controlnet.py) and [Checkpoint](https://huggingface.co/PixArt-alpha/PixArt-ControlNet/tree/main) are released!!!\n- (🔥 New) Jan. 16, 2024. 💥 Glad to announce that [PixArt-α](https://arxiv.org/abs/2310.00426) is accepted by ICLR 2024 (Spotlight).\n- (🔥 New) Dec. 17, 2023. 💥 PixArt supports [ComfyUI](https://github.com/comfyanonymous/ComfyUI#manual-install-windows-linux). Thanks to [@city96](https://github.com/city96/ComfyUI_ExtraModels) with his great work.\n- (🔥 New) Nov. 30, 2023. 💥 PixArt collaborates with [LCMs](https://github.com/luosiallen/latent-consistency-model) team to make the **fastest** [Training \u0026 Inference Text-to-Image Generation System](https://github.com/PixArt-alpha/PixArt-alpha).\nHere, [Training code](train_scripts/train_pixart_lcm.py) \u0026 [Inference code](scripts/inference_lcm.py) \u0026 [Weights](https://huggingface.co/PixArt-alpha/PixArt-LCM-XL-2-1024-MS) \u0026 [HF Demo](https://huggingface.co/spaces/PixArt-alpha/PixArt-LCM) [OpenXLab Demo](https://openxlab.org.cn/apps/detail/houshaowei/PixArt-LCM) are all released, we hope users will enjoy them. \nDetailed **inference speed** and **code guidance** can be found in [docs](asset/docs/pixart_lcm.md). At the same time, we update the codebase for better user experience and fix some bugs in the newest version.\n\n---\n## 🚩 **New Features/Updates**\n- ✅ Jan. 11, 2024. 💥 [PixArt-δ](https://arxiv.org/abs/2401.05252): We are excited to announce the release of the [PixArt-δ](https://arxiv.org/abs/2401.05252) technical report!!!\nThis report offers valuable insights into the training of LCM and ControlNet-like modules in Transformer Models. Along with the report, we have also released all the training and inference code for LCM \u0026 ControlNet [in this repository](https://github.com/PixArt-alpha/PixArt-alpha). \nWe encourage you to try them out and warmly welcome any Pull Requests from our users. Your contributions and feedback are highly appreciated!\n- ✅ Feb. 07, 2024. [train_diffusers.py](train_scripts/train_diffusers.py) can directly train with diffusers model and visualize during training.\n- ✅ Jan. 26, 2024. 💥 All checkpoints of [PixArt-α](https://github.com/PixArt-alpha/PixArt-alpha), including 256px checkpoints are all available here [Download Models](#-download-models).\n- ✅ Jan. 19, 2024. 💥 [PixArt-δ](https://arxiv.org/abs/2401.05252) ControlNet [app_controlnet.py](app/app_controlnet.py) and [Checkpoint](https://huggingface.co/PixArt-alpha/PixArt-ControlNet/tree/main) is released!!!\n- ✅ Jan. 12, 2024. 💥 We release the [SAM-LLaVA-Captions](https://huggingface.co/datasets/PixArt-alpha/SAM-LLaVA-Captions10M) used in PixArt-α training.\n- ✅ Dec. 27, 2023. [PixArt-α](https://github.com/PixArt-alpha/PixArt-alpha) incorporates into [ControlLLM](https://github.com/OpenGVLab/ControlLLM)!\n- ✅ Dec. 17, 2023. [PixArt-LCM-Lora](train_scripts/train_pixart_lcm_lora.py) \u0026 [PixArt-Lora](train_scripts/train_pixart_lora_hf.py) training scripts in Hugging Face style is released.\n- ✅ Dec. 13, 2023. Add multi-scale vae feature extraction in [tools/extract_features.py](https://github.com/PixArt-alpha/PixArt-alpha/blob/3b4f0afdbe39def80b41ab05c664c963edeebbcd/tools/extract_features.py#L276).\n- ✅ Dec. 01, 2023. Add a [Notebook folder](./notebooks) to help users get started with PixArt quickly! Thanks to [@kopyl](https://github.com/kopyl) for his contribution!\n- ✅ Nov. 27, 2023. 💥 **PixArt-α Community**: Join our PixArt-α discord channels \u003ca href=\"https://discord.gg/rde6eaE5Ta\" style=\"text-decoration:none;\"\u003e\n\u003cimg src=\"https://user-images.githubusercontent.com/25839884/218347213-c080267f-cbb6-443e-8532-8e1ed9a58ea9.png\" width=\"3%\" alt=\"\" /\u003e\u003c/a\u003e for discussions. Coders are welcome to contribute.\n- ✅ Nov. 21, 2023. 💥 [SA-Sovler](https://arxiv.org/abs/2309.05019) official code first release [here](asset/docs/sasolver.md).\n- ✅ Nov. 19, 2023. Release `PixArt + Dreambooth` training scripts.\n- ✅ Nov. 16, 2023. Diffusers support `random resolution` and `batch images` generation now. Besides, \nrunning `Pixart` in under 8GB GPU VRAM is available in 🧨 [diffusers](https://huggingface.co/docs/diffusers/main/en/api/pipelines/pixart).\n- ✅ Nov. 10, 2023. Support DALL-E 3 Consistency Decoder in 🧨 diffusers.\n- ✅ Nov. 06, 2023. Release pretrained weights with 🧨 diffusers integration, Hugging Face demo, and Google Colab example.\n- ✅ Nov. 03, 2023. Release the LLaVA-captioning inference code.\n- ✅ Oct. 27, 2023. Release the training \u0026 feature extraction code.\n- ✅ Oct. 20, 2023. Collaborate with Hugging Face \u0026 Diffusers team to co-release the code and weights. (plz stay tuned.)\n- ✅ Oct. 15, 2023. Release the inference code.\n\n---\n\n## Contents\n* [Training](#-how-to-train)\n* [Inference](#-how-to-test)\n* [Download Models](#-download-models)\n* [Use diffusers](#1---using-in--diffusers)\n* [Data Processing](#-how-to-extract-t5-and-vae-features)\n* [PixArt-**α** Demo](#3---gradio-with-diffusers--faster-)\n* [PixArt-**α** 8GB VRAM](asset/docs/pixart.md)\n* [PixArt-**δ** (LCM)](asset/docs/pixart_lcm.md)\n* [PixArt-**δ** (ControlNet)](asset/docs/pixart_controlnet.md)\n* [PixArt-**δ** (Dreambooth)](asset/docs/pixart-dreambooth.md)\n* [Acknowledgement](#acknowledgements)\n* [Citation](#bibtex)\n\n\n* [PixArt-**Σ** Releasing](https://github.com/PixArt-alpha/PixArt-sigma)\n\n---\n\n## 🐱 Abstract\n\u003cb\u003eTL; DR: \u003cfont color=\"red\"\u003ePixArt-α\u003c/font\u003e is a Transformer-based T2I diffusion model whose image generation quality is competitive with state-of-the-art image generators (e.g., Imagen, SDXL, and even Midjourney), and the training speed markedly surpasses existing large-scale T2I models, e.g., PixArt-α only takes 10.8% of Stable Diffusion v1.5's training time (675 vs. 6,250 A100 GPU days).\u003c/b\u003e\n\n\u003cdetails\u003e\u003csummary\u003eCLICK for the full abstract\u003c/summary\u003e\nThe most advanced text-to-image (T2I) models require significant training costs (e.g., millions of GPU hours), \nseriously hindering the fundamental innovation for the AIGC community while increasing CO2 emissions. \nThis paper introduces PixArt-α, a Transformer-based T2I diffusion model whose image generation quality is competitive with state-of-the-art image generators (e.g., Imagen, SDXL, and even Midjourney), \nreaching near-commercial application standards. Additionally, it supports high-resolution image synthesis up to 1024px resolution with low training cost. \nTo achieve this goal, three core designs are proposed: \n(1) Training strategy decomposition: We devise three distinct training steps that separately optimize pixel dependency, text-image alignment, and image aesthetic quality; \n(2) Efficient T2I Transformer: We incorporate cross-attention modules into Diffusion Transformer (DiT) to inject text conditions and streamline the computation-intensive class-condition branch; \n(3) High-informative data: We emphasize the significance of concept density in text-image pairs and leverage a large Vision-Language model to auto-label dense pseudo-captions to assist text-image alignment learning. \nAs a result, PixArt-α's training speed markedly surpasses existing large-scale T2I models, \ne.g., PixArt-α only takes 10.8% of Stable Diffusion v1.5's training time (675 vs. 6,250 A100 GPU days), \nsaving nearly $300,000 ($26,000 vs. $320,000) and reducing 90% CO2 emissions. Moreover, compared with a larger SOTA model, RAPHAEL, \nour training cost is merely 1%. Extensive experiments demonstrate that PixArt-α excels in image quality, artistry, and semantic control. \nWe hope PixArt-α will provide new insights to the AIGC community and startups to accelerate building their own high-quality yet low-cost generative models from scratch.\n\u003c/details\u003e\n\n---\n\n![A small cactus with a happy face in the Sahara desert.](asset/images/teaser.png)\n\n---\n\n# 🔥🔥🔥 Why PixArt-α? \n## Training Efficiency\nPixArt-α only takes 12% of Stable Diffusion v1.5's training time (753 vs. 6,250 A100 GPU days), saving nearly $300,000 ($28,000 vs. $320,000) and reducing 90% CO2 emissions. Moreover, compared with a larger SOTA model, RAPHAEL, our training cost is merely 1%.\n![Training Efficiency.](asset/images/efficiency.png)\n\n| Method    | Type | #Params | #Images| FID-30K ↓        | A100 GPU days |\n|-----------|------|---------|--------|------------------|---------------|\n| DALL·E    | Diff | 12.0B   | 250M   | 27.50            |               |\n| GLIDE     | Diff | 5.0B    | 250M   | 12.24            |               |\n| LDM       | Diff | 1.4B    | 400M   | 12.64            |               |\n| DALL·E 2  | Diff | 6.5B    | 650M   | 10.39            | 41,66         |\n| SDv1.5    | Diff | 0.9B    | 2000M  | 9.62             | 6,250         |\n| GigaGAN   | GAN  | 0.9B    | 2700M  | 9.09             | 4,783         |\n| Imagen    | Diff | 3.0B    | 860M   | 7.27             | 7,132         |\n| RAPHAEL   | Diff | 3.0B    | 5000M+ | 6.61             | 60,000        |\n| PixArt-α  | Diff | 0.6B    | 25M    | 7.32 (zero-shot) | 753           |\n| PixArt-α  | Diff | 0.6B    | 25M    | 5.51 (COCO FT)   | 753           |\n\n## Inference Efficiency\nPIXART-δ successfully generates **1024x1024 high resolution** images within **0.5 seconds** on an A100. With the implementation\nof 8-bit inference technology, PIXART-δ requires **less than 8GB of GPU VRAM**. \n\nLet us stress again how liberating it is to explore image generation so easily with PixArt-LCM.\n\n| Hardware                    | PIXART-δ (4 steps) | SDXL LoRA LCM (4 steps) | PixArt-α (14 steps) | SDXL standard (25 steps) |\n|-----------------------------|--------------------|-------------------------|---------------------|---------------------------|\n| T4 (Google Colab Free Tier) | 3.3s               | 8.4s                    | 16.0s               | 26.5s                     |\n| V100 (32 GB)                | 0.8s               | 1.2s                    | 5.5s                | 7.7s                      |\n| A100 (80 GB)                | 0.51s              | 1.2s                    | 2.2s                | 3.8s                      |\n\nThese tests were run with a batch size of 1 in all cases.\n\nFor cards with a lot of capacity, such as A100, performance increases significantly when generating multiple images at once, which is usually the case for production workloads.\n\n## High-quality Generation from PixArt-α\n\n- More samples\n\u003cdiv id=\"more-samples\" style=\"display: flex; justify-content: center;\"\u003e\n  \u003cimg src=\"asset/images/more-samples1.png\" style=\"width: 50%; height: auto; object-fit: contain; margin: 5px;\"\u003e\n  \u003cimg src=\"asset/images/more-samples.png\" style=\"width: 43%; height: auto; object-fit: contain; margin: 5px;\"\u003e\n\u003c/div\u003e\n\n- PixArt + [Dreambooth](https://dreambooth.github.io/)\n\u003cdiv id=\"dreambooth\" style=\"display: flex; justify-content: center;\"\u003e\n  \u003cimg src=\"asset/images/dreambooth/dreambooth_dog.svg\" width=\"46%\" style=\"margin: 5px;\"\u003e\n  \u003cimg src=\"asset/images/dreambooth/dreambooth_m5.svg\" width=\"46%\" style=\"margin: 5px;\"\u003e\n\u003c/div\u003e\n\n- PixArt + [ControlNet](https://github.com/lllyasviel/ControlNet)\n\u003cdiv id=\"ControlNet\" style=\"display: flex; justify-content: center;\"\u003e\n  \u003cimg src=\"asset/images/controlnet/controlnet_huawei.svg\" width=\"46%\" style=\"margin: 5px;\"\u003e\n  \u003cimg src=\"asset/images/controlnet/controlnet_lenna.svg\" width=\"46%\" style=\"margin: 5px;\"\u003e\n\u003c/div\u003e\n\n# 🔧 Dependencies and Installation\n\n- Python \u003e= 3.9 (Recommend to use [Anaconda](https://www.anaconda.com/download/#linux) or [Miniconda](https://docs.conda.io/en/latest/miniconda.html))\n- [PyTorch \u003e= 1.13.0+cu11.7](https://pytorch.org/)\n```bash\nconda create -n pixart python=3.9\nconda activate pixart\npip install torch==2.1.1 torchvision==0.16.1 torchaudio==2.1.1 --index-url https://download.pytorch.org/whl/cu118\n\ngit clone https://github.com/PixArt-alpha/PixArt-alpha.git\ncd PixArt-alpha\npip install -r requirements.txt\n```\n\n# ⏬ Download Models\nAll models will be automatically downloaded. You can also choose to download manually from this [url](https://huggingface.co/PixArt-alpha/PixArt-alpha).\n\n| Model                       | #Params | url                                                                                                                                                                                                          | Download in OpenXLab                                                                                            |\n|:----------------------------|:--------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------|\n| T5                          | 4.3B    | [T5](https://huggingface.co/PixArt-alpha/PixArt-alpha/tree/main/t5-v1_1-xxl)                                                                                                                                 | [T5](https://download.openxlab.org.cn/models/PixArt-alpha/PixArt-alpha/weight/t5-v1_1-xxl.zip)                  |\n| VAE                         | 80M     | [VAE](https://huggingface.co/PixArt-alpha/PixArt-alpha/tree/main/sd-vae-ft-ema)                                                                                                                              | [VAE](https://download.openxlab.org.cn/models/PixArt-alpha/PixArt-alpha/weight/sd-vae-ft-ema.zip)               |\n| PixArt-α-SAM-256            | 0.6B    | [PixArt-XL-2-SAM-256x256.pth](https://huggingface.co/PixArt-alpha/PixArt-alpha/resolve/main/PixArt-XL-2-SAM-256x256.pth) or [diffusers version](https://huggingface.co/PixArt-alpha/PixArt-XL-2-SAM-256x256) | [256-SAM](https://download.openxlab.org.cn/models/PixArt-alpha/PixArt-alpha/weight/PixArt-XL-2-SAM-256x256.pth) |\n| PixArt-α-256                | 0.6B    | [PixArt-XL-2-256x256.pth](https://huggingface.co/PixArt-alpha/PixArt-alpha/resolve/main/PixArt-XL-2-256x256.pth) or [diffusers version](https://huggingface.co/PixArt-alpha/PixArt-XL-2-256x256)             | [256](https://download.openxlab.org.cn/models/PixArt-alpha/PixArt-alpha/weight/PixArt-XL-2-256x256.pth)         |\n| PixArt-α-256-MSCOCO-FID7.32 | 0.6B    | [PixArt-XL-2-256x256.pth](https://huggingface.co/PixArt-alpha/PixArt-alpha/resolve/main/PixArt-XL-2-256x256-MSCOCO-FID732.pth)                                                                               | [256]()                                                                                                         |\n| PixArt-α-512                | 0.6B    | [PixArt-XL-2-512x512.pth](https://huggingface.co/PixArt-alpha/PixArt-alpha/resolve/main/PixArt-XL-2-512x512.pth) or [diffusers version](https://huggingface.co/PixArt-alpha/PixArt-XL-2-512x512)             | [512](https://download.openxlab.org.cn/models/PixArt-alpha/PixArt-alpha/weight/PixArt-XL-2-512x512.pth)         |\n| PixArt-α-1024               | 0.6B    | [PixArt-XL-2-1024-MS.pth](https://huggingface.co/PixArt-alpha/PixArt-alpha/resolve/main/PixArt-XL-2-1024-MS.pth) or [diffusers version](https://huggingface.co/PixArt-alpha/PixArt-XL-2-1024-MS)             | [1024](https://download.openxlab.org.cn/models/PixArt-alpha/PixArt-alpha/weight/PixArt-XL-2-1024-MS.pth)        |\n| PixArt-δ-1024-LCM           | 0.6B    | [diffusers version](https://huggingface.co/PixArt-alpha/PixArt-LCM-XL-2-1024-MS)                                                                                                                             |                                                                                                                 |\n| ControlNet-HED-Encoder      | 30M     | [ControlNetHED.pth](https://huggingface.co/PixArt-alpha/PixArt-alpha/blob/main/ControlNetHED.pth)                                                                                                            |                                                                                                                 |\n| PixArt-δ-512-ControlNet     | 0.9B    | [PixArt-XL-2-512-ControlNet.pth](https://huggingface.co/PixArt-alpha/PixArt-ControlNet/tree/main)                                                                                                            | [512](https://openxlab.org.cn/models/detail/PixArt-alpha/PixArt-ControlNet)                                     |\n| PixArt-δ-1024-ControlNet    | 0.9B    | [PixArt-XL-2-1024-ControlNet.pth](https://huggingface.co/PixArt-alpha/PixArt-ControlNet/tree/main)                                                                                                           | [1024](https://openxlab.org.cn/models/detail/PixArt-alpha/PixArt-ControlNet)                                    |\n\nALSO find all models in [OpenXLab_PixArt-alpha](https://openxlab.org.cn/models/detail/PixArt-alpha/PixArt-alpha)\n\n# 🔥 How to Train\n## 1. PixArt Training\n\n**First of all.**\n\nThanks to [@kopyl](https://github.com/kopyl), you can reproduce the full fine-tune training flow on [Pokemon dataset](https://huggingface.co/datasets/lambdalabs/pokemon-blip-captions) from HugginFace with notebooks:\n1. Train with [notebooks/train.ipynb](https://github.com/PixArt-alpha/PixArt-alpha/blob/53dac066f60fe5fdbdde4f0360145ca96d4cc38c/notebooks/train.ipynb).\n2. Convert to Diffusers with [notebooks/convert-checkpoint-to-diffusers.ipynb](https://github.com/PixArt-alpha/PixArt-alpha/blob/master/notebooks/convert-checkpoint-to-diffusers.ipynb).\n3. Run the inference with converted checkpoint in step 2 with [notebooks/infer.ipynb](https://github.com/PixArt-alpha/PixArt-alpha/blob/master/notebooks/infer.ipynb).\n\n**Then, for more details.**\n\nHere we take SAM dataset training config as an example, but of course, you can also prepare your own dataset following this method.\n\nYou **ONLY** need to change the **config** file in [config](./configs/pixart_config) and **dataloader** in [dataset](./diffusion/data/datasets).\n```bash\npython -m torch.distributed.launch --nproc_per_node=2 --master_port=12345 train_scripts/train.py configs/pixart_config/PixArt_xl2_img256_SAM.py --work-dir output/train_SAM_256\n```\n\nThe directory structure for SAM dataset is:\n```\ncd ./data\n\nSA1B\n├──images/  (images are saved here)\n│  ├──sa_xxxxx.jpg\n│  ├──sa_xxxxx.jpg\n│  ├──......\n├──captions/    (corresponding captions are saved here, same name as images)\n│  ├──sa_xxxxx.txt\n│  ├──sa_xxxxx.txt\n├──partition/   (all image names are stored txt file where each line is a image name)\n│  ├──part0.txt\n│  ├──part1.txt\n│  ├──......\n├──caption_feature_wmask/   (run tools/extract_caption_feature.py to generate caption T5 features, same name as images except .npz extension)\n│  ├──sa_xxxxx.npz\n│  ├──sa_xxxxx.npz\n│  ├──......\n├──img_vae_feature/  (run tools/extract_img_vae_feature.py to generate image VAE features, same name as images except .npy extension)\n│  ├──train_vae_256/\n│  │  ├──noflip/\n│  │  │  ├──sa_xxxxx.npy\n│  │  │  ├──sa_xxxxx.npy\n│  │  │  ├──......\n\n```\n\n**Here we prepare data_toy for better understanding**\n```bash\ncd ./data\n\ngit lfs install\ngit clone https://huggingface.co/datasets/PixArt-alpha/data_toy\n```\nThen, \n[Here](https://huggingface.co/datasets/PixArt-alpha/data_toy/blob/main/part0.txt) is an example of partition/part0.txt file.\n\n---\n\nBesides, for json file guided [training](https://github.com/PixArt-alpha/PixArt-alpha/blob/fe0cb78065d64c18ecd8955a04e4f29138d47946/configs/pixart_config/PixArt_xl2_img1024_internalms.py#L3C2-L3C2),\n[here](https://huggingface.co/datasets/PixArt-alpha/data_toy/blob/main/data_info.json) is a toy json file for better understand.\n\n---\n\n## 2. PixArt + DreamBooth Training\n\nFollowing the `Pixart + DreamBooth` [training guidance](asset/docs/pixart-dreambooth.md)\n\n## 3. PixArt + LCM / LCM-LoRA Training\n\nFollowing the `PixArt + LCM` [training guidance](asset/docs/pixart_lcm.md)\n\n## 4. PixArt + ControlNet Training\n\nFollowing the `PixArt + ControlNet` [training guidance](asset/docs/pixart_controlnet.md)\n\n## 4. PixArt + LoRA Training\n\n```bash\npip install peft==0.6.2\n\naccelerate launch --num_processes=1 --main_process_port=36667  train_scripts/train_pixart_lora_hf.py --mixed_precision=\"fp16\" \\\n  --pretrained_model_name_or_path=PixArt-alpha/PixArt-XL-2-1024-MS \\\n  --dataset_name=lambdalabs/pokemon-blip-captions --caption_column=\"text\" \\\n  --resolution=1024 --random_flip \\\n  --train_batch_size=16 \\\n  --num_train_epochs=200 --checkpointing_steps=100 \\\n  --learning_rate=1e-06 --lr_scheduler=\"constant\" --lr_warmup_steps=0 \\\n  --seed=42 \\\n  --output_dir=\"pixart-pokemon-model\" \\\n  --validation_prompt=\"cute dragon creature\" --report_to=\"tensorboard\" \\\n  --gradient_checkpointing --checkpoints_total_limit=10 --validation_epochs=5 \\\n  --rank=16\n```\n\n# 💻 How to Test\nInference requires at least `23GB` of GPU memory using this repo, while `11GB and 8GB` using in 🧨 [diffusers](#using-in--diffusers).\n\nCurrently support:\n- [x] [IDDPM](https://arxiv.org/abs/2102.09672)\n- [x] [DPM-Solver](https://arxiv.org/abs/2206.00927)\n- [x] [SA-Solver](https://arxiv.org/abs/2309.05019)\n- [ ] [DPM-Solver-v3](https://arxiv.org/abs/2310.13268v2)\n\n## 1. Quick start with [Gradio](https://www.gradio.app/guides/quickstart)\n\nTo get started, first install the required dependencies. Make sure you've downloaded the [models](https://huggingface.co/PixArt-alpha/PixArt-alpha) to the output/pretrained_models folder, and then run on your local machine:\n\n```bash\nDEMO_PORT=12345 python app/app.py\n```\n\nAs an alternative, a sample [Dockerfile](Dockerfile) is provided to make a runtime container that starts the Gradio app.\n\n```bash\ndocker build . -t pixart\ndocker run --gpus all -it -p 12345:12345 -v \u003cpath_to_huggingface_cache\u003e:/root/.cache/huggingface pixart\n```\n\nOr use docker-compose.  Note, if you want to change context from the 1024 to 512 or LCM version of the app just change the APP_CONTEXT env variable in the docker-compose.yml file.  The default is 1024\n\n```bash\ndocker compose build\ndocker compose up\n```\n\nLet's have a look at a simple example using the `http://your-server-ip:12345`.\n\n\n## 2. Integration in diffusers\n### 1). Using in 🧨 diffusers\n\nMake sure you have the updated versions of the following libraries:\n\n```bash\npip install -U transformers accelerate diffusers SentencePiece ftfy beautifulsoup4\n```\n\nAnd then:\n\n```python\nimport torch\nfrom diffusers import PixArtAlphaPipeline, ConsistencyDecoderVAE, AutoencoderKL\ndevice = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n\n# You can replace the checkpoint id with \"PixArt-alpha/PixArt-XL-2-512x512\" too.\npipe = PixArtAlphaPipeline.from_pretrained(\"PixArt-alpha/PixArt-XL-2-1024-MS\", torch_dtype=torch.float16, use_safetensors=True)\n\n# If use DALL-E 3 Consistency Decoder\n# pipe.vae = ConsistencyDecoderVAE.from_pretrained(\"openai/consistency-decoder\", torch_dtype=torch.float16)\n\n# If use SA-Solver sampler\n# from diffusion.sa_solver_diffusers import SASolverScheduler\n# pipe.scheduler = SASolverScheduler.from_config(pipe.scheduler.config, algorithm_type='data_prediction')\n\n# If loading a LoRA model\n# transformer = Transformer2DModel.from_pretrained(\"PixArt-alpha/PixArt-LCM-XL-2-1024-MS\", subfolder=\"transformer\", torch_dtype=torch.float16)\n# transformer = PeftModel.from_pretrained(transformer, \"Your-LoRA-Model-Path\")\n# pipe = PixArtAlphaPipeline.from_pretrained(\"PixArt-alpha/PixArt-LCM-XL-2-1024-MS\", transformer=transformer, torch_dtype=torch.float16, use_safetensors=True)\n# del transformer\n\n# Enable memory optimizations.\n# pipe.enable_model_cpu_offload()\n\npipe.to(device)\n\nprompt = \"A small cactus with a happy face in the Sahara desert.\"\nimage = pipe(prompt).images[0]\nimage.save(\"./catcus.png\")\n```\nCheck out the [documentation](./asset/docs/sasolver.md) for more information about SA-Solver Sampler.\n\nThis integration allows running the pipeline with a batch size of 4 under 11 GBs of GPU VRAM. \nCheck out the [documentation](https://huggingface.co/docs/diffusers/main/en/api/pipelines/pixart) to learn more.\n\n### 2). Running the `PixArtAlphaPipeline` in under 8GB GPU VRAM\n\nGPU VRAM consumption under 8 GB is supported now, please refer to [documentation](asset/docs/pixart.md) for more information.\n\n### 3). Gradio with diffusers (Faster)\n\nTo get started, first install the required dependencies, then run on your local machine:\n\n```bash\n# diffusers version\nDEMO_PORT=12345 python app/app.py\n```\nLet's have a look at a simple example using the `http://your-server-ip:12345`.\n\nYou can also click [here](https://colab.research.google.com/drive/1jZ5UZXk7tcpTfVwnX33dDuefNMcnW9ME?usp=sharing) to have a free trial on Google Colab.\n\n### 4). Convert .pth checkpoint into diffusers version\n\n```bash\npython tools/convert_pixart_alpha_to_diffusers.py --image_size your_img_size --multi_scale_train (True if you use PixArtMS else False) --orig_ckpt_path path/to/pth --dump_path path/to/diffusers --only_transformer=True\n```\n\n\n## 3. Online Demo [![Hugging Face PixArt](https://img.shields.io/static/v1?label=Demo\u0026message=HuggingFace%20Gradio\u0026color=orange)](https://huggingface.co/spaces/PixArt-alpha/PixArt-alpha) \n![Online Demo sample](asset/images/sample.png)\n\n# ✏️ How to LLaVA captioning\nThanks to the code base of [LLaVA-Lightning-MPT](https://huggingface.co/liuhaotian/LLaVA-Lightning-MPT-7B-preview), \nwe can caption the LAION and SAM dataset with the following launching code:\n```bash\npython tools/VLM_caption_lightning.py --output output/dir/ --data-root data/root/path --index path/to/data.json\n```\nWe present auto-labeling with custom prompts for LAION (left) and SAM (right). The words highlighted in green represent the original caption in LAION, while those marked in red indicate the detailed captions labeled by LLaVA.\n\n![Dialog with LLaVA.](asset/images/LLaVA-dialog.png)\n\n# ✏️ How to extract T5 and VAE features\n\nPrepare T5 text feature and VAE image feature in advance will speed up the training process and save GPU memory.\n```bash\npython tools/extract_features.py --img_size=1024 \\\n    --json_path \"data/data_info.json\" \\\n    --t5_save_root \"data/SA1B/caption_feature_wmask\" \\\n    --vae_save_root \"data/SA1B/img_vae_features\" \\\n    --pretrained_models_dir \"output/pretrained_models\" \\\n    --dataset_root \"data/SA1B/Images/\"\n```\n\n## 💪To-Do List (Congratulations🎉)\n\n- [x] Inference code\n- [x] Training code\n- [x] T5 \u0026 VAE feature extraction code\n- [x] LLaVA captioning code\n- [x] Model zoo \n- [x] Diffusers version \u0026 Hugging Face demo\n- [x] Google Colab example\n- [x] DALLE3 VAE integration\n- [x] Inference under 8GB GPU VRAM with diffusers\n- [x] Dreambooth Training code\n- [x] SA-Solver code\n- [x] PixArt-α-LCM will release soon\n- [x] Multi-scale vae feature extraction code\n- [x] PixArt-α-LCM-LoRA scripts will release soon\n- [x] PixArt-α-LoRA training scripts will release soon\n- [x] ControlNet code will be released\n- [x] SAM-LLaVA caption dataset\n- [x] ControlNet checkpoint\n- [x] 256px pre-trained models\n- [x] PixArt-Σ: Next version model with much better ability is training!\n\n# Other Source\nWe make a video comparing PixArt with current most powerful Text-to-Image models.\n\n[![Watch the video](https://img.youtube.com/vi/7_6KsIITgWY/maxresdefault.jpg)](https://www.youtube.com/watch?v=7_6KsIITgWY)\n\n# 📖BibTeX\n    @misc{chen2023pixartalpha,\n          title={PixArt-$\\alpha$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis}, \n          author={Junsong Chen and Jincheng Yu and Chongjian Ge and Lewei Yao and Enze Xie and Yue Wu and Zhongdao Wang and James Kwok and Ping Luo and Huchuan Lu and Zhenguo Li},\n          year={2023},\n          eprint={2310.00426},\n          archivePrefix={arXiv},\n          primaryClass={cs.CV}\n    }\n    @misc{chen2024pixartdelta,\n          title={PIXART-{\\delta}: Fast and Controllable Image Generation with Latent Consistency Models}, \n          author={Junsong Chen and Yue Wu and Simian Luo and Enze Xie and Sayak Paul and Ping Luo and Hang Zhao and Zhenguo Li},\n          year={2024},\n          eprint={2401.05252},\n          archivePrefix={arXiv},\n          primaryClass={cs.CV}\n    }\n    \n# 🤗Acknowledgements\n- Thanks to [Diffusers](https://github.com/huggingface/diffusers) for their wonderful technical support and awesome collaboration!\n- Thanks to [Hugging Face](https://github.com/huggingface) for sponsoring the nicely demo!\n- Thanks to [DiT](https://github.com/facebookresearch/DiT) for their wonderful work and codebase!\n\n## Star History\n\n[![Star History Chart](https://api.star-history.com/svg?repos=PixArt-alpha/PixArt-alpha\u0026type=Date)](https://star-history.com/#PixArt-alpha/PixArt-alpha\u0026Date)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPixArt-alpha%2FPixArt-alpha","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FPixArt-alpha%2FPixArt-alpha","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPixArt-alpha%2FPixArt-alpha/lists"}