{"id":27943627,"url":"https://github.com/alpha-vllm/lumina-mgpt-2.0","last_synced_at":"2025-05-07T12:18:13.565Z","repository":{"id":285930731,"uuid":"959591157","full_name":"Alpha-VLLM/Lumina-mGPT-2.0","owner":"Alpha-VLLM","description":"Lumina-mGPT 2.0: Stand-alone Autoregressive Image Modeling","archived":false,"fork":false,"pushed_at":"2025-05-01T12:40:50.000Z","size":19321,"stargazers_count":681,"open_issues_count":2,"forks_count":40,"subscribers_count":15,"default_branch":"main","last_synced_at":"2025-05-07T12:18:05.227Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Alpha-VLLM.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-04-03T03:14:13.000Z","updated_at":"2025-05-07T09:05:21.000Z","dependencies_parsed_at":"2025-04-03T12:42:41.805Z","dependency_job_id":null,"html_url":"https://github.com/Alpha-VLLM/Lumina-mGPT-2.0","commit_stats":null,"previous_names":["alpha-vllm/lumina-mgpt-2.0"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alpha-VLLM%2FLumina-mGPT-2.0","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alpha-VLLM%2FLumina-mGPT-2.0/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alpha-VLLM%2FLumina-mGPT-2.0/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alpha-VLLM%2FLumina-mGPT-2.0/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Alpha-VLLM","download_url":"https://codeload.github.com/Alpha-VLLM/Lumina-mGPT-2.0/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252873889,"owners_count":21817715,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-07T12:18:12.691Z","updated_at":"2025-05-07T12:18:13.413Z","avatar_url":"https://github.com/Alpha-VLLM.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n \u003cimg src=\"./assets/lumina-mgpt-2.0.png\" width=\"15%\"/\u003e\n \u003cbr\u003e\n\u003c/p\u003e\n\n\u003cdiv align=\"center\"\u003e\n\u003ch1\u003e Lumina-mGPT 2.0: Stand-alone Autoregressive Image Modeling \u003c/h1\u003e\n\n\u003cb\u003e¹Shanghai AI Laboratory, \u0026nbsp;  ²SII, \u0026nbsp; ³CUHK, \u0026nbsp;⁴ZJUT\u003c/b\u003e\n\n\u003c/div\u003e\n\n## 📚 Introduction \nWe introduce a stand-alone, decoder-only autoregressive model, **trained from scratch**, that unifies a broad spectrum of image generation tasks, including **text-to-image generation, image pair generation, subject-driven generation, multi-turn image editing, controllable generation, and dense prediction**.\n\n\u003cdetails open\u003e\n  \u003csummary\u003eUser Demo\u003c/summary\u003e\n  \u003cvideo src=\"https://github.com/user-attachments/assets/7b87eeac-3f20-450c-8df0-a45218acc51d\" width=\"100%\"/\u003e\u003c/video\u003e\n\u003c/details\u003e\n   \n\u003cdetails open\u003e\n  \u003csummary\u003eArchitecture\u003c/summary\u003e\n  \u003cimg src=\"./assets/architecture.png\" width=\"100%\"/\u003e\n\u003c/details\u003e\n\n\n\n## 🔥 News\n**[2025-04-03]** 🎉🎉🎉 Lumina-mGPT 2.0 is released! 🎉🎉🎉\n\n\n## 📝 Open-source Plan\n - [x] Text-to-Image / Image Pair Generation Inference \u0026 Checkpoints\n - [x] Finetuning code\n - [ ] All-in-One Inference \u0026 Checkpoints\n - [ ] Technical Report\n\n## 📽️ Demo Examples\n\u003cdetails open\u003e\n  \u003csummary\u003eQualitative Performance\u003c/summary\u003e\n \u003cimg src=\"./assets/qualitative.jpg\" width=\"100%\"/\u003e\n\u003c/details\u003e\n\n\u003cdetails open\u003e\n  \u003csummary\u003eComparison with Lumina-mGPT and Janus Pro\u003c/summary\u003e\n  \u003cimg src=\"./assets/comparison.png\" width=\"100%\"/\u003e\n\u003c/details\u003e\n\n\n## 🚀 Quick Start\n### ⚙️ Installation\n#### 1. Create a conda environment\n```\ngit clone https://github.com/Alpha-VLLM/Lumina-mGPT-2.0.git \u0026\u0026 cd Lumina-mGPT-2.0\nconda create -n lumina_mgpt_2 python=3.10 -y\nconda activate lumina_mgpt_2\n```\n#### 2. Install  dependencies\n```\npip install -r requirements.txt\npip install https://github.com/Dao-AILab/flash-attention/releases/download/v2.7.4.post1/flash_attn-2.7.4.post1+cu12torch2.3cxx11abiFALSE-cp310-cp310-linux_x86_64.whl --no-build-isolation\npip install -e .\n```\n\u003e Kindly find proper flash-attn version from this [link](https://github.com/Dao-AILab/flash-attention/releases).\n#### 3. Download MoVQGAN\nDownload MoVQGAN weights and put them to the ```lumina_mgpt/movqgan/270M/movqgan_270M.ckpt```.\n```\nmkdir -p lumina_mgpt/movqgan/270M\nwget -O lumina_mgpt/movqgan/270M/movqgan_270M.ckpt https://huggingface.co/ai-forever/MoVQGAN/resolve/main/movqgan_270M.ckpt\n```\n\n### ⛽ Inference\n#### 1. Simple Inference\n```\npython generate_examples/generate.py \\\n--model_path Alpha-VLLM/Lumina-mGPT-2.0 --save_path save_samples/ \\\n--cfg 4.0 --top_k 4096 --temperature 1.0 --width 768 --height 768\n```\n\n#### 2. Accelerate Inference\nProvide two acceleration strategies: Speculative Jacobi Decoding (```--speculative_jacobi```) and Model Quantization (```--quant```).\n```\npython generate_examples/generate.py \\\n--model_path Alpha-VLLM/Lumina-mGPT-2.0 --save_path save_samples/ \\\n--cfg 4.0 --top_k 4096 --temperature 1.0 --width 768 --height 768 \\\n--speculative_jacobi --quant\n```\n\nWe provide the inference time and GPU memory on one A100 as a reference:\n| Method               | Inference Time | Inference GPU Memory | Description |\n|----------------------|--------|--------|--------------------|\n| Lumina-mGPT 2.0      | 694s   | 80 GB  | ✅ Recommend |\n| + speculative_jacobi | 324s     | 79.2 GB  | ✅ Recommend |\n| + speculative_jacobi \u0026 quant | 304s     | 33.8 GB  |  |\n\n### 💻 Finetuning\nPlease refer to  [TRAIN.md](TRAIN.md)\n\n\n## 🤗 Checkpoints\n| Model                | Size | Reso | pth link | Description |\n|----------------------|--------|--------|--------------------|--------------------|\n| Lumina-mGPT 2.0      | 7B     | 768px  | [7B_768px](https://huggingface.co/Alpha-VLLM/Lumina-mGPT-2.0)       | Text-to-Image \u0026 Image Pair Generation |\n\n## 📜 Acknowledgements\n\n**Thanks to the following open-sourced codebase for their wonderful work and codebase!**\n- [Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining](https://github.com/Alpha-VLLM/Lumina-mGPT)\n- [Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding](https://github.com/tyshiwo1/Accelerating-T2I-AR-with-SJD/)\n- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://github.com/facebookresearch/chameleon)\n\n\n## 🔥 Open Positions\nWe are hiring interns and full-time researchers at the Alpha VLLM Group, Shanghai AI Lab. If you are interested, please contact alphavllm@gmail.com.\n\n## 🌟 Star History\n\n[![Star History Chart](https://api.star-history.com/svg?repos=Alpha-VLLM/Lumina-mGPT-2.0\u0026type=Date)](https://www.star-history.com/#Alpha-VLLM/Lumina-mGPT-2.0\u0026Date)\n\n## 📖 BibTeX\n\n```\n@misc{lumina-mgpt-2.0,\n      title={Lumina-mGPT 2.0: Stand-alone Autoregressive Image Modeling},\n      author={Alpha VLLM Team},\n      year={2025},\n      url={https://github.com/Alpha-VLLM/Lumina-mGPT-2.0},\n}\n```\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falpha-vllm%2Flumina-mgpt-2.0","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falpha-vllm%2Flumina-mgpt-2.0","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falpha-vllm%2Flumina-mgpt-2.0/lists"}