{"id":27943622,"url":"https://github.com/alpha-vllm/lumina-accessory","last_synced_at":"2025-08-25T12:25:57.947Z","repository":{"id":289062952,"uuid":"969993561","full_name":"Alpha-VLLM/Lumina-Accessory","owner":"Alpha-VLLM","description":null,"archived":false,"fork":false,"pushed_at":"2025-04-25T08:39:04.000Z","size":18758,"stargazers_count":85,"open_issues_count":4,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-05-07T12:18:05.326Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Alpha-VLLM.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-21T09:26:25.000Z","updated_at":"2025-05-07T02:51:54.000Z","dependencies_parsed_at":"2025-04-21T10:56:00.887Z","dependency_job_id":null,"html_url":"https://github.com/Alpha-VLLM/Lumina-Accessory","commit_stats":null,"previous_names":["alpha-vllm/lumina-accessory"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alpha-VLLM%2FLumina-Accessory","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alpha-VLLM%2FLumina-Accessory/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alpha-VLLM%2FLumina-Accessory/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alpha-VLLM%2FLumina-Accessory/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Alpha-VLLM","download_url":"https://codeload.github.com/Alpha-VLLM/Lumina-Accessory/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252873889,"owners_count":21817715,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-07T12:18:12.361Z","updated_at":"2025-05-07T12:18:12.902Z","avatar_url":"https://github.com/Alpha-VLLM.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003ch1\u003e Lumina-Accessory: Instruction Fine-tuned Rectified Flow Transformer for Universial Image Generation \u003c/h1\u003e\n\n[![Lumina-Next](https://img.shields.io/badge/Paper-Lumina--Image--2.0-2b9348.svg?logo=arXiv)](https://arxiv.org/abs/2503.21758)\u0026#160;\n[![Badge](https://img.shields.io/badge/-WeChat@Join%20Our%20Group-000000?logo=wechat\u0026logoColor=07C160)](https://github.com/ChinChyi/ipictures/blob/main/20250421.jpg?raw=true)\u0026#160;\n[![Static Badge](https://img.shields.io/badge/Lumina--Accessory%20checkpoints-Model(2B)-yellow?logoColor=violet\u0026label=%F0%9F%A4%97%20Lumina-Accessory%20checkpoints)](https://huggingface.co/Alpha-VLLM/Lumina-Accessory)\n\n\u003cp align=\"center\"\u003e\n \u003cimg src=\"./assets/Illustration.png\" width=\"100%\"/\u003e\n \u003cbr\u003e\n\u003c/p\u003e\n\n\u003c/div\u003e\n\u003cdiv align=\"center\"\u003e\n\n\u003c/div\u003e\n\n## ✨ Features\n\n**Lumina-Accessory** is a multi-task instruction fine-tuning framework designed for Lumina series (currently supporting **Lumina-Image-2.0**). This repository includes:  \n\n- **🧠 Tuning Code** – Unifies various image-to-image tasks in a **sequence concatenation manner**, supporting both **universal** and **task-specific** model tuning.  \n\n- **⚖️ Instruction Fine-tuned Universal Model Weights** – Initialized from **Lumina-Image-2.0**, supporting:  \n  - 🖼️ **Spatial conditional generation**  \n  - 🔧 **Infilling \u0026 Restoration**  \n  - 💡 **Relighting**  \n  - 🎨 **Subject-driven generation**  \n  - ✏️ **Instruction-based editing**  \n\n- **🚀 Inference Code \u0026 Gradio Demo** – Test and showcase the **universal model’s capabilities** interactively!  \n\n## 📰 News\n- [2025-4-21] 🚀🚀🚀 We are excited to release `Lumina-Accessory`, including:\n  - 🎯 Checkpoints, Fine-Tuning and Inference code.\n\n## 📑 Open-source Plan\n\n - [x] Tuning code\n - [x] Inference Code\n - [x] Checkpoints\n - [ ] Web Demo (Gradio)\n\n## 🏠 Architecture\n✨ Lumina-Accessory directly leverages the self-attention mechanism in DiT to perform interaction between condition and target image tokens, consistent with approaches such as [OminiControl](https://github.com/Yuanshi9815/OminiControl), [DSD](https://primecai.github.io/dsd/), [VisualCloze](https://github.com/lzyhha/VisualCloze), etc.\n\n✨ Built on top of Lumina-Image-2.0, Lumina-Accessory introduces an additional condition processor, initialized with the weights of the latent processor.\n\n✨ Similar to [OminiControl](https://github.com/Yuanshi9815/OminiControl), we modulate both condition and target image tokens with different time conditions, and apply distinct positional embeddings for different types of conditions.\n\n\u003cp align=\"center\"\u003e\n \u003cimg src=\"./assets/Architecture.png\" width=\"100%\"/\u003e\n \u003cbr\u003e\n\u003c/p\u003e\n\n## 🎮 Model Zoo\n\n| Resolution | Parameter| Text Encoder | VAE | Download URL  |\n| ---------- | ----------------------- | ------------ | -----------|-------------- |\n| 1024       | 2.6B             |    [Gemma-2-2B](https://huggingface.co/google/gemma-2-2b)  |   [FLUX-VAE-16CH](https://huggingface.co/black-forest-labs/FLUX.1-dev/tree/main/vae) | [hugging face](https://huggingface.co/Alpha-VLLM/Lumina-Accessory) |\n\n## 📊 Model Capability\n\n| Task Type                  | Training Data               | Model Ability               |\n|----------------------------|-----------------------------|-----------------------------|\n| **Spatial Conditional Generation** | Internal Data    | 😄 (Good)           |\n| **Infilling \u0026 Restoration** | Internal Data     | 😄 (Good)          |\n| **Relighting**             | IC-Light Synthetic Data | 😊 (Moderate)            |\n| **Subject-Driven Generation** | Subject200K | 😐 (Basic)                 |\n| **Instruction-Based Editing** | OmniEdit-1.2M    | 😐 (Basic)                 |\n\n## 💻 Finetuning Code\n### 1. Create a conda environment and install PyTorch\n```bash\nconda create -n Lumina2 -y\nconda activate Lumina2\nconda install python=3.11 pytorch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 pytorch-cuda=12.1 -c pytorch -c nvidia -y\n```\n### 2.Install dependencies\n```bash\npip install -r requirements.txt\n```\n### 3. Install flash-attn\n```bash\npip install flash-attn --no-build-isolation\n```\n### 4. Prepare data\nYou can place the links to your data files in `./configs/data.yaml`. \n\nFor tasks where the condition can be generated online, your image-text pair training data format should adhere to the following:\n```json\n{\n    \"image_path\": \"path/to/your/image\",\n    \"prompt\": \"a description of the image\"\n}\n```\n\nFor tasks that require loading a condition image, the training data format should be as follows:\n```json\n{\n    \"input_image\": \"path/to/your/condition\",\n    \"output_image\": \"path/to/your/target\",\n    \"prompt\": \"a description of the image\"\n}\n```\n### 5. Start finetuning\n```bash\nbash scripts/run_1024_finetune.sh\n```\n## 🚀 Inference Code\nWe support multiple solvers including Midpoint Solver, Euler Solver, and **DPM Solver** for inference.\n\u003e [!Note]\n\u003e Both the Gradio demo and the direct inference method use the .pth format weight file, which can be downloaded from [huggingface](https://huggingface.co/Alpha-VLLM/Lumina-Accessory). We have uploaded the .pth weight files, and you can simply specify the `--ckpt` argument as the download directory.\n\n\u003e [!Note]\n\u003e The code has just been cleaned up, if there are any issues please let us know.\n\n- Direct Inference\n```python   \nNUM_STEPS=50\nCFG_SCALE=4.0\nTIME_SHIFTING_FACTOR=6\nSEED=20\nSOLVER=euler\nTASK_TYPE=\"Image Infilling\"\nCAP_DIR=./examples/caption_list.json\nOUT_DIR=./examples/outputs\nMODEL_CHECKPOINT=/path/to/your/ckpt\n\npython -u sample_accessory.py --ckpt ${MODEL_CHECKPOINT} \\\n--image_save_path ${OUT_DIR} \\\n--solver ${SOLVER} \\\n--num_sampling_steps ${STEPS} \\\n--caption_path ${CAP_DIR} \\\n--seed ${SEED} \\\n--time_shifting_factor ${TIME_SHIFTING_FACTOR} \\\n--cfg_scale ${CFG_SCALE} \\\n--batch_size 1 \\\n--rank 0 \\\n--task_type \"${TASK_TYPE}\"\n```\n\n- Gradio Demo\n```python   \nPRECISION=\"bf16\" \nSOLVER=\"euler\"\nVAE=\"flux\"\nSHARE=False\nMODEL_CHECKPOINT=/path/to/your/ckpt\n\ntorchrun --nproc_per_node=1 --master_port=18187 gradio_demo.py \\\n  --ckpt \"$MODEL_CHECKPOINT\" \\\n  --precision \"$PRECISION\" \\\n  --solver \"$SOLVER\" \\\n  --vae \"$VAE\" \\\n  --share \"$SHARE\"\n```\n\n\u003cp align=\"left\"\u003e\n \u003cimg src=\"./assets/Demo.png\" width=\"70%\"/\u003e\n \u003cbr\u003e\n\u003c/p\u003e\n\n## Citation\n\nIf you find the provided code or models useful for your research, consider citing them as:\n\n```bib\n@Misc{lumina-accessory,\n  author = {Alpha-VLLM Team},\n  title  = {Lumina-Accessory GitHub Page},\n  year   = {2025},\n}\n```\n\n## Related Work\n[Lumina-Image 2.0: A Unified and Efficient Image Generative Framework](https://github.com/Alpha-VLLM/Lumina-Image-2.0)\n\n[OminiControl: Minimal and Universal Control for Diffusion Transformer](https://github.com/Yuanshi9815/OminiControl)\n\n[Diffusion Self-Distillation for Zero-Shot Customized Image Generation](https://primecai.github.io/dsd/)\n\n[OmniEdit: Building Image Editing Generalist Models Through Specialist Supervision](https://tiger-ai-lab.github.io/OmniEdit/)\n\n[VisualCloze: A Universal Image Generation Framework via Visual In-Context Learning](https://github.com/lzyhha/VisualCloze)\n\n[OminiControl2: Efficient Conditioning for Diffusion Transformers](https://arxiv.org/abs/2503.08280)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falpha-vllm%2Flumina-accessory","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falpha-vllm%2Flumina-accessory","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falpha-vllm%2Flumina-accessory/lists"}