{"id":27943629,"url":"https://github.com/alpha-vllm/lumina-mgpt","last_synced_at":"2025-05-15T22:10:50.070Z","repository":{"id":251525228,"uuid":"837095937","full_name":"Alpha-VLLM/Lumina-mGPT","owner":"Alpha-VLLM","description":"Official Implementation of \"Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining\"","archived":false,"fork":false,"pushed_at":"2025-04-01T12:49:47.000Z","size":11206,"stargazers_count":594,"open_issues_count":17,"forks_count":28,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-05-15T22:09:58.540Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2408.02657","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Alpha-VLLM.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-08-02T07:50:23.000Z","updated_at":"2025-05-08T15:16:55.000Z","dependencies_parsed_at":"2024-08-16T18:32:47.831Z","dependency_job_id":"c351b720-d03f-43e2-927c-9d82b81147e5","html_url":"https://github.com/Alpha-VLLM/Lumina-mGPT","commit_stats":null,"previous_names":["alpha-vllm/lumina-mgpt"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alpha-VLLM%2FLumina-mGPT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alpha-VLLM%2FLumina-mGPT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alpha-VLLM%2FLumina-mGPT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alpha-VLLM%2FLumina-mGPT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Alpha-VLLM","download_url":"https://codeload.github.com/Alpha-VLLM/Lumina-mGPT/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254430331,"owners_count":22069909,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-07T12:18:12.717Z","updated_at":"2025-05-15T22:10:45.054Z","avatar_url":"https://github.com/Alpha-VLLM.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n\u003cimg src=\"assets/logo.png\" width=\"30%\"/\u003e\n\n# Lumina-mGPT\n\n\u003cb\u003e A family of multimodal autoregressive models capable of various vision and language tasks, particularly excelling in generating flexible photorealistic images from text descriptions. 👋 join our \u003ca href=\"http://imagebind-llm.opengvlab.com/qrcode/\" target=\"_blank\"\u003eWeChat\u003c/a\u003e \u003c/b\u003e\n\n[![Lumina-mGPT](https://img.shields.io/badge/Paper-Lumina--mGPT-2b9348.svg?logo=arXiv)](https://arxiv.org/abs/2408.02657)\u0026#160;\n\n[![Static Badge](https://img.shields.io/badge/Official(node1)-6B88E3?logo=youtubegaming\u0026label=Demo%20Lumina-mGPT)](http://106.14.2.150:10020/)\u0026#160;\n[![Static Badge](https://img.shields.io/badge/Official(node2)-6B88E3?logo=youtubegaming\u0026label=Demo%20Lumina-mGPT)](http://106.14.2.150:10021/)\u0026#160;\n\n\u003c/div\u003e\n\n\u003cimg src=\"assets/demos.png\"\u003e\n\n## 📰 News\n\n- **[2024-08-11] 🎉🎉🎉 [Training codes and documents](./lumina_mgpt/TRAIN.md) are released! 🎉🎉🎉**\n\n- **[2024-07-08] 🎉🎉🎉 Lumina-mGPT is released! 🎉🎉🎉**\n\n## ⚙️ Installation\n\nSee [INSTALL.md](./INSTALL.md) for detailed instructions.\n\nNote that the Lumina-mGPT implementation heavily relies on\nthe [xllmx](./xllmx) module, which is evolved from [LLaMA2-Accessory](https://github.com/Alpha-VLLM/LLaMA2-Accessory) for supporting\nLLM-centered multimodal tasks. Make sure it is installed correctly as a python package before going on.\n\n## ⛽ Training\nSee [lumina_mgpt/TRAIN.md](lumina_mgpt/TRAIN.md)\n\n## 📽️ Inference\n\n\u003e [!Note]\n\u003e\n\u003e Before using the Lumina-mGPT model, run\n\u003e\n\u003e ```bash\n\u003e # bash\n\u003e cd lumina_mgpt\n\u003e ```\n\u003e\n\u003e to enter the directory of the Lumina-mGPT implementation.\n\n### Perpetration\n\nSince currently the Chameleon implementation in transformers does not contain the VQ-VAE decoder, please manually download the original VQ-VAE weights [provided by Meta](https://github.com/facebookresearch/chameleon) and\nput them to the following directory:\n\n```\nLumina-mGPT\n- lumina_mgpt/\n    - ckpts/\n        - chameleon/\n            - tokenizer/\n                - text_tokenizer.json\n                - vqgan.yaml\n                - vqgan.ckpt\n- xllmx/\n- ...\n```\n\n### Local Gradio Demos\n\nWe have prepared three different Gradio demos, each showcasing unique functionalities, to help you quickly become familiar with the capabilities of the Lumina-mGPT models.\n\n#### 1. [demos/demo_image_generation.py](./Lumina-mGPT/demos/demo_image_generation.py)\n\nThis demo is customized for Image Generation tasks, where you can input a text description and generate a corresponding image.\nTo host this demo, run:\n\n```bash\n# Note to set the `--target_size` argument consistent with the checkpoint\npython -u demos/demo_image_generation.py \\\n--pretrained_path Alpha-VLLM/Lumina-mGPT-7B-768 \\\n--target_size 768\n```\n\n#### 2. [demos/demo_image2image.py](./Lumina-mGPT/demos/demo_image2image.py)\n\nThis demo is designed for models trained with Omni-SFT. you can conveniently switch between the multiple downstream tasks using this demo.\n\n```bash\n# Note to set the `--target_size` argument consistent with the checkpoint\npython -u demos/demo_image2image.py \\\n--pretrained_path Alpha-VLLM/Lumina-mGPT-7B-768-Omni \\\n--target_size 768\n```\n\n#### 3. [demos/demo_freeform.py](./Lumina-mGPT/demos/demo_freeform.py)\n\nThis is a powerful demo with minimal constraint on the input format. It supports flexible interation and is suitable for in-deep exploration.\n\n```bash\n# Note to set the `--target_size` argument consistent with the checkpoint\npython -u demos/demo_freeform.py \\\n--pretrained_path Alpha-VLLM/Lumina-mGPT-7B-768-Omni \\\n--target_size 768\n```\n\n### Simple Inference\n\nThe simplest code for Lumina-mGPT inference:\n\n```python\nfrom inference_solver import FlexARInferenceSolver\nfrom PIL import Image\n\n# ******************** Image Generation ********************\ninference_solver = FlexARInferenceSolver(\n    model_path=\"Alpha-VLLM/Lumina-mGPT-7B-768\",\n    precision=\"bf16\",\n    target_size=768,\n)\n\nq1 = f\"Generate an image of 768x768 according to the following prompt:\\n\"\n     f\"Image of a dog playing water, and a waterfall is in the background.\"\n\n# generated: tuple of (generated response, list of generated images)\ngenerated = inference_solver.generate(\n    images=[],\n    qas=[[q1, None]],\n    max_gen_len=8192,\n    temperature=1.0,\n    logits_processor=inference_solver.create_logits_processor(cfg=4.0, image_top_k=2000),\n)\n\na1, new_image = generated[0], generated[1][0]\n\n\n# ******************* Image Understanding ******************\ninference_solver = FlexARInferenceSolver(\n    model_path=\"Alpha-VLLM/Lumina-mGPT-7B-512\",\n    precision=\"bf16\",\n    target_size=512,\n)\n\n# \"\u003c|image|\u003e\" symbol will be replaced with sequence of image tokens before fed to LLM\nq1 = \"Describe the image in detail. \u003c|image|\u003e\"\n\nimages = [Image.open(\"image.png\")]\nqas = [[q1, None]]\n\n# `len(images)` should be equal to the number of appearance of \"\u003c|image|\u003e\" in qas\ngenerated = inference_solver.generate(\n    images=images,\n    qas=qas,\n    max_gen_len=8192,\n    temperature=1.0,\n    logits_processor=inference_solver.create_logits_processor(cfg=4.0, image_top_k=2000),\n)\n\na1 = generated[0]\n# generated[1], namely the list of newly generated images, should typically be empty in this case.\n\n\n# ********************* Omni-Potent *********************\ninference_solver = FlexARInferenceSolver(\n    model_path=\"Alpha-VLLM/Lumina-mGPT-7B-768-Omni\",\n    precision=\"bf16\",\n    target_size=768,\n)\n\n# Example: Depth Estimation\n# For more instructions, see demos/demo_image2image.py\nq1 = \"Depth estimation. \u003c|image|\u003e\"\nimages = [Image.open(\"image.png\")]\nqas = [[q1, None]]\n\ngenerated = inference_solver.generate(\n    images=images,\n    qas=qas,\n    max_gen_len=8192,\n    temperature=1.0,\n    logits_processor=inference_solver.create_logits_processor(cfg=1.0, image_top_k=200),\n)\n\na1 = generated[0]\nnew_image = generated[1][0]\n\n```\n\n## 🤗 Checkpoints\n\n**Configurations**\n\n\u003cimg src=\"assets/config2.jpg\"\u003e\n\u003cimg src=\"assets/config1.jpg\"\u003e\n\n**7B models**\n\n| Model        | Size | Huggingface                                                                              |\n| ------------ | ---- | ---------------------------------------------------------------------------------------- |\n| FP-SFT@512   | 7B   | [Alpha-VLLM/Lumina-mGPT-7B-512](https://huggingface.co/Alpha-VLLM/Lumina-mGPT-7B-512)       |\n| FP-SFT@768   | 7B   | [Alpha-VLLM/Lumina-mGPT-7B-768](https://huggingface.co/Alpha-VLLM/Lumina-mGPT-7B-768)       |\n| Omni-SFT@768 | 7B   | [Alpha-VLLM/Lumina-mGPT-7B-768-Omni](https://huggingface.co/Alpha-VLLM/Lumina-mGPT-7B-768-Omni) |\n| FP-SFT@1024  | 7B   | [Alpha-VLLM/Lumina-mGPT-7B-1024](https://huggingface.co/Alpha-VLLM/Lumina-mGPT-7B-1024)     |\n\n**34B models**\n\n| Model      | Size | Huggingface                                                                          |\n| ---------- | ---- | ------------------------------------------------------------------------------------ |\n| FP-SFT@512 | 34B  | [Alpha-VLLM/Lumina-mGPT-34B-512](https://huggingface.co/Alpha-VLLM/Lumina-mGPT-34B-512) |\n\nMore checkpoints coming soon.\n\n## 📑 Open-source Plan\n\n- [X] Inference code\n- [X] Training code\n\n## 🔥 Open positions\nWe are hiring interns, postdocs, and full-time researchers at the General Vision Group, Shanghai AI Lab, with a focus on multi-modality and vision foundation models. If you are interested, please contact gaopengcuhk@gmail.com.\n\n## 📄 Citation\n\n```\n@misc{liu2024lumina-mgpt,\n      title={Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining},\n      author={Dongyang Liu and Shitian Zhao and Le Zhuo and Weifeng Lin and Yu Qiao and Hongsheng Li and Peng Gao},\n      year={2024},\n      eprint={2408.02657},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV},\n      url={https://arxiv.org/abs/2408.02657},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falpha-vllm%2Flumina-mgpt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falpha-vllm%2Flumina-mgpt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falpha-vllm%2Flumina-mgpt/lists"}