{"id":29219672,"url":"https://github.com/dvlab-research/llmga","last_synced_at":"2025-07-03T02:06:37.996Z","repository":{"id":209518436,"uuid":"724274335","full_name":"dvlab-research/LLMGA","owner":"dvlab-research","description":"This project is the official implementation of 'LLMGA: Multimodal Large Language Model based Generation Assistant', ECCV2024  Oral","archived":false,"fork":false,"pushed_at":"2025-06-02T14:05:39.000Z","size":15802,"stargazers_count":395,"open_issues_count":3,"forks_count":24,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-06-03T02:57:08.354Z","etag":null,"topics":["aigc","image-design-assistant","image-editing","image-generation","large-language-model","llm","mllm","multi-modal"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dvlab-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-11-27T18:46:55.000Z","updated_at":"2025-06-02T14:05:46.000Z","dependencies_parsed_at":"2023-11-29T02:28:12.850Z","dependency_job_id":"41448866-a53d-400b-9990-dc3f38d89b7c","html_url":"https://github.com/dvlab-research/LLMGA","commit_stats":null,"previous_names":["zj-binxia/llmga","dvlab-research/llmga"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dvlab-research/LLMGA","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dvlab-research%2FLLMGA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dvlab-research%2FLLMGA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dvlab-research%2FLLMGA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dvlab-research%2FLLMGA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dvlab-research","download_url":"https://codeload.github.com/dvlab-research/LLMGA/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dvlab-research%2FLLMGA/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263245318,"owners_count":23436514,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aigc","image-design-assistant","image-editing","image-generation","large-language-model","llm","mllm","multi-modal"],"created_at":"2025-07-03T02:06:35.144Z","updated_at":"2025-07-03T02:06:37.977Z","avatar_url":"https://github.com/dvlab-research.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\n\n\u003cp align=\"center\" width=\"10%\"\u003e\n\u003cimg src=\"imgs/logo.png\" style=\"width: 30%\" align=center\u003e \n\u003c/p\u003e\n\n\n# LLMGA: Multimodal Large Language Model-based Generation Assistant (ECCV2024  Oral)\n\n[Bin Xia](https://scholar.google.com/citations?user=rh2fID8AAAAJ\u0026hl=zh-CN), [Shiyin Wang](), [Yingfan Tao](https://scholar.google.com/citations?user=GYDnPdQAAAAJ\u0026hl=zh-CN\u0026oi=ao), [Yitong Wang](https://scholar.google.com/citations?user=NfFTKfYAAAAJ\u0026hl=zh-CN), and [Jiaya Jia](https://scholar.google.com/citations?user=XPAkzTEAAAAJ\u0026hl=zh-CN\u0026oi=ao)\n\n\u003ca href=\"https://llmga.github.io/\"\u003e\u003cimg src=\"https://img.shields.io/badge/Project-Page-Green\"\u003e\u003c/a\u003e\n\u003ca href=\"https://arxiv.org/pdf/2311.16500.pdf\"\u003e\u003cimg src='https://img.shields.io/badge/Paper-Arxiv-red'\u003e\u003c/a\u003e \n\u003ca href='https://huggingface.co/binxia'\u003e\u003cimg src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Models-blue'\u003e\u003c/a\u003e\n\u003ca href='https://huggingface.co/datasets/binxia/LLMGA-datasetv2/tree/main'\u003e\u003cimg src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Data-green'\u003e\u003c/a\u003e\n\n## News\n**New Version (Accepted by ECCV2024):**\n- [x] [2024.07.06] The finetuned SD15 models have been released, including [SD15-T2I](https://huggingface.co/binxia/llmga-sd15-t2i-v2) and [SD15-inpainting](https://huggingface.co/binxia/llmga-sd15-inpainting-v2). Notably, our SD15-T2I model can also be used for instruction-based editing of LLMGA.\n- [x] [2024.07.06] The finetuned SDXL models have been released, including [SDXL-T2I](https://huggingface.co/binxia/llmga-sdxl-t2i) and [SDXL-inpainting](https://huggingface.co/binxia/llmga-sdxl-inpainting-v2/tree/main). \n- [x] [2024.07.06] The pre-trained models, which further support Chinese (obtained by further fine-tuned on mixed Chinese and English data),  have been released, including [llmga-cn-vicuna 7b](https://huggingface.co/binxia/llmga-cn-vicuna-7b-v1.5-full-finetune), [llmga-cn-llama3 8b](https://huggingface.co/binxia/llmga-cn-llama3-8b-it-full-finetune), [llmga-cn-gemma 2b](https://huggingface.co/binxia/llmga-cn-gemma-2b-it-full-finetune), and [llmga-cn-qwen2 0.5b](https://huggingface.co/binxia/llmga-cn-Qwen2-0.5B-full-finetune).\n- [x] [2024.07.06] We release new version LLMGA's [training datasets](https://huggingface.co/datasets/binxia/LLMGA-datasetv2/tree/main), including texts and images.\n- [x] [2024.07.05] The pre-trained model has been released, including [llmga-vicuna 7b](https://huggingface.co/binxia/llmga-vicuna-7b-v1.5-full-finetune/tree/main), [llmga-mistral 7b](https://huggingface.co/binxia/llmga-mistral_instruct-full-finetune/tree/main), [llmga-llama3 8b](https://huggingface.co/binxia/llmga-llama3-8b-it-full-finetune/tree/main), [llmga-vicuna7b](https://huggingface.co/binxia/llmga-vicuna-7b-v1.5-full-finetune/tree/main), [llmga-qwen2 0.5b](https://huggingface.co/binxia/llmga-Qwen2-0.5B-full-finetune/tree/main), [llmga-qwen2 1.5b](https://huggingface.co/binxia/llmga-Qwen2-1.5B-full-finetune/tree/main), [llmga-qwen2 7b](https://huggingface.co/binxia/llmga-Qwen2-7B-full-finetune/tree/main), [llmga-phi3 3b](https://huggingface.co/binxia/llmga-Phi-3-mini-128k-full-finetune/tree/main), and [llmga-gemma 2b](https://huggingface.co/binxia/llmga-gemma-2b-it-full-finetune/tree/main).\n- [x] [2024.07.05] The code has been updated.\n- [x] [2024.07.04] I am organizing and uploading the new version of the LLMGA code and the dataset and model. I will have a status update when I complete this process, please wait for me for a few days. Notably, in this new version, we build our LLMGA on different base LLM models, such as **Llama2 7b**, **Mistral 7b**, **LLama3 8b**, **Qwen2 0.5b**, **Qwen2 1.5b**, **Qwen2 7b**, **Phi3 3b**, and **gemma 2b**. They have different performance and model sizes, as well as commercial licenses, there is always one that can meet your usage scenario.\n      \n**Old Version:**\n- [x] [2023.12.20]   We release LLMGA's [training datasets].\n- [x] [2023.12.20]    We release the gradio codes of LLMGA7b-SDXL-T2I.\n- [x] [2023.12.08]   We release LLMGA7b-SDXL-T2I [demo].\n- [x] [2023.11.30]   We have released the code for DiffRIR. It can effectively eliminate differences in brightness, contrast, and texture between generated and preserved regions in inpainting and outpainting. Considering its applicability to projects beyond LLMGA, we have open-sourced it at [Github](https://github.com/Zj-BinXia/DiffRIR).\n- [x] [2023.11.29]   The models is released at [Huggingface].\n- [x] [2023.11.29]   The training and inference code is released.\n- [x] [2023.11.29]  We will upload all models, code, and data within a week and further refine this project.\n- [x] [2023.11.28]    GitHub repo is created.\n\n---\n\n\u003e **Abstract:** In this paper, we introduce a Multimodal Large Language Model-based Generation Assistant (LLMGA), leveraging the vast reservoir of knowledge and proficiency in reasoning, comprehension, and response inherent in Large Language Models (LLMs) to assist users in image generation and editing. Diverging from existing approaches where Multimodal Large Language Models (MLLMs) generate fixed-size embeddings to control Stable Diffusion (SD), our LLMGA provides a detailed language generation prompt for precise control over SD. This not only augments LLM context understanding but also reduces noise in generation prompts, yields images with more intricate and precise content, and elevates the interpretability of the network. To this end, we curate a comprehensive dataset comprising prompt refinement, similar image generation, inpainting \\\u0026 outpainting, and instruction-based editing. Moreover, we propose a two-stage training scheme. In the first stage, we train the MLLM to grasp the properties of image generation and editing, enabling it to generate detailed prompts. In the second stage, we optimize SD to align with the MLLM's generation prompts. Additionally, we propose a reference-based restoration network to alleviate texture, brightness, and contrast disparities between generated and preserved regions during inpainting and outpainting. Extensive results show that LLMGA has promising generation and editing capabilities and can enable more flexible and expansive applications in an interactive manner.\n\n---\n\n## Why do you need LLMGA?\n\n- [x] **Generation Assiatant**. As a unified system, LLMGA can generate and edit images using methods such as Text-to-Image (T2I), inpainting, outpainting, and instruction-based editing through conversational interactions with users. By leveraging the extensive knowledge and understanding of image design from LLMGA, users can easily produce and revise images to obtain highly satisfactory images.\n- [x]  **Design Expert**. LLMGA incorporates an extensive array of image design data, offering deep insights for a wide range of design tasks, including logo creation, game character design, poster design, T-shirt design, infographic design, and more.\n- [x]  **Illustration Generation**. LLMGA can interactively generate story illustrations based on user-input story snippets.\n- [x]  **Picture Book Generation**. With a single user's instruction, LLMGA can generate an interwoven storybook of text and illustrations.\n- [x]  **Multilingual Support**.Through the multilingual adaptation of the LLMGA, T2I and editing model can generate content using Chinese language instructions.\n- [x]  **Flexible Expansion**. LLMGA offers enhanced flexibility by integrating with external plugins like ControlNet, enabling a wider range of functionalities.\n- [x] To be continued ......\n\n\u003cdiv align=center\u003e\n\u003cimg width=\"100%\" src=\"imgs/github_poster1.png\"/\u003e\n\u003c/div\u003e\n\n\u003cdiv align=center\u003e\n\u003cimg width=\"100%\" src=\"imgs/demo1.png\"/\u003e\n\u003c/div\u003e\n\n\u003cdiv align=center\u003e\n\u003cimg width=\"100%\" src=\"imgs/demo2.png\"/\u003e\n\u003c/div\u003e\n\n\n## Contents\n- [TODO](#todo)\n- [Install](#install)\n- [Model](#model)\n- [Preparation](#Preparation)\n- [Train](#train)\n- [Inference](#inference)\n- [Citation](#citation)\n- [Acknowledgement](#acknowledgement)\n\n## TODO\n- [x] Support gradio demo.\n- [ ] Support more generation models\n\n\n\n## Install\nPlease follow the instructions below to install the required packages.\n1. Clone this repository\n```bash\ngit clone https://github.com/dvlab-research/LLMGA.git\n```\n\n2. Install Package\n```bash\nconda create -n llmga python=3.9 -y\nconda activate llmga\ncd LLMGA\npip install --upgrade pip  # enable PEP 660 support\npip install -e .\ncd ./llmga/diffusers\npip install . \n```\n\n3. Install additional packages for training cases\n```bash\npip install -e \".[train]\"\npip install -r requirements.txt\npip install flash-attn --no-build-isolation\npip install datasets\npip install albumentations\npip install ninja\n```\n\n\n\n## Model\n\n\n\u003cdiv align=center\u003e\n\u003cimg width=\"100%\" src=\"imgs/method.png\"/\u003e\n\u003c/div\u003e\n\n\n\n## Preparation\n\n### Training Dataset\nWe provide the training data for LLMGA training. \n\nplease download [LLMGA datasets](https://huggingface.co/datasets/binxia/LLMGA-datasetv2) and [LLaVA pretrain datasets](https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain). \n\nBesides, download LLaVA1.5 instruction tuning datasets [llava_v1_5_mix665k.json](https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K/blob/main/llava_v1_5_mix665k.json), and download the images from constituting datasets:\n- COCO: [train2017](http://images.cocodataset.org/zips/train2017.zip)\n- GQA: [images](https://downloads.cs.stanford.edu/nlp/data/gqa/images.zip)\n- OCR-VQA: [download script](https://drive.google.com/drive/folders/1_GYPY5UkUy7HIcR0zq3ZCFgeZN7BAfm_?usp=sharing), **we save all files as `.jpg`**\n- TextVQA: [train_val_images](https://dl.fbaipublicfiles.com/textvqa/images/train_val_images.zip)\n- VisualGenome: [part1](https://cs.stanford.edu/people/rak248/VG_100K_2/images.zip), [part2](https://cs.stanford.edu/people/rak248/VG_100K_2/images2.zip). \n\nPlease organize these downloaded data as in [Structure](#structure).\n\n### The MLP Projector Pretrained Weights\nWe recommend users to download the [pretrained MLP projector weights](https://huggingface.co/binxia/LLMGA-pretrained-mlp/tree/main). Then put them in `./checkpoints` following [Structure](#structure).\n\n\n### Inference Pretrained Weights\n\nPlease download MLLM Models and SD models from the following links. For example, you can download [LLMGA-MLLM7b](https://huggingface.co/binxia/llmga-llama-2-7b-chat-full-finetune) and [LLMGA-SDXL-T2I](https://huggingface.co/binxia/llmga-sdxl-t2i) to realize LLMGA7b-T2I functionality. Please organize them as in [Structure](#structure).\n\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003cth align=\"left\"\u003eMLLM Model (support English)\u003c/th\u003e\n    \u003cth align=\"center\"\u003ePretrained Models\u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"left\"\u003ellmga-vicuna 7b\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://huggingface.co/binxia/llmga-vicuna-7b-v1.5-full-finetune/tree/main\"\u003eDownload\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"left\"\u003ellmga-mistral 7b\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://huggingface.co/binxia/llmga-mistral_instruct-full-finetune/tree/main\"\u003eDownload\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"left\"\u003ellmga-llama3 8b\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://huggingface.co/binxia/llmga-llama3-8b-it-full-finetune/tree/main\"\u003eDownload\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"left\"\u003ellmga-qwen2 0.5b\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://huggingface.co/binxia/llmga-Qwen2-0.5B-full-finetune/tree/main\"\u003eDownload\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"left\"\u003ellmga-qwen2 1.5b\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://huggingface.co/binxia/llmga-Qwen2-1.5B-full-finetune/tree/main\"\u003eDownload\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"left\"\u003ellmga-qwen2 7b\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://huggingface.co/binxia/llmga-Qwen2-7B-full-finetune/tree/main\"\u003eDownload\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"left\"\u003ellmga-phi3 3b\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://huggingface.co/binxia/llmga-Phi-3-mini-128k-full-finetune/tree/main\"\u003eDownload\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"left\"\u003ellmga-gemma 2b\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://huggingface.co/binxia/llmga-gemma-2b-it-full-finetune/tree/main\"\u003eDownload\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003cth align=\"left\"\u003eMLLM Model (further support Chinese and English)\u003c/th\u003e\n    \u003cth align=\"center\"\u003ePretrained Models\u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"left\"\u003ellmga-cn-vicuna 7b\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://huggingface.co/binxia/llmga-cn-vicuna-7b-v1.5-full-finetune\"\u003eDownload\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"left\"\u003ellmga-cn-llama3 8b\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://huggingface.co/binxia/llmga-cn-llama3-8b-it-full-finetune\"\u003eDownload\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"left\"\u003ellmga-cn-gemma 2b\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://huggingface.co/binxia/llmga-cn-gemma-2b-it-full-finetune\"\u003eDownload\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"left\"\u003ellmga-cn-qwen2 0.5b\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://huggingface.co/binxia/llmga-cn-Qwen2-0.5B-full-finetune\"\u003eDownload\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003cth align=\"left\"\u003eSD Model\u003c/th\u003e\n    \u003cth align=\"center\"\u003ePretrained Models\u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"left\"\u003eLLMGA-SD15-T2I\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://huggingface.co/binxia/llmga-sd15-t2i-v2\"\u003eDownload\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"left\"\u003eLLMGA-SD15-Inpainting\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://huggingface.co/binxia/llmga-sd15-inpainting-v2\"\u003eDownload\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"left\"\u003eLLMGA-SDXL-T2I\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://huggingface.co/binxia/llmga-sdxl-t2i\"\u003eDownload\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"left\"\u003eLLMGA-SDXL-Inpainting\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003ca href=\"https://huggingface.co/binxia/llmga-sdxl-inpainting-v2\"\u003eDownload\u003c/a\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n### Structure\n\nThe folder structure should be organized as follows before training.\n\n```\nLLMGA\n├── llmga\n├── scripts\n├── work_dirs\n├── checkpoints\n│   ├── llmga-Phi-3-mini-128k-pretrain\n│   ├── llmga-Qwen2-0.5B-pretrain\n│   ├── llmga-llama3-8b-pretrain\n│   ├── llmga-mistral-pretrain\n│   ├── llmga-vicuna-7b-v1.5-pretrain\n│   ├── llmga-Phi-3-mini-128k-full-finetune\n│   ├── llmga-Qwen2-0.5B-full-finetune\n│   ├── llmga-llama3-8b-it-full-finetune\n│   ├── llmga-mistral_instruct-full-finetune\n│   ├── llmga-vicuna-7b-v1.5-full-finetune\n│   ├── llmga-cn-vicuna-7b-v1.5-full-finetune\n│   ├── llmga-cn-Qwen2-0.5B-full-finetune\n│   ├── llmga-sdxl-t2i\n│   ├── llmga-sd15-inpainting-v2\n│   ├── llmga-sd15-t2i-v2\n├── data\n│   │── jsons\n│   │   ├── llmga-data\n│   │   │   ├── Edit/train.json\n│   │   │   ├── inpainting/train.json\n│   │   │   ├── SG/train.json\n│   │   │   ├── T2I/train.json\n│   │   ├── text-data\n│   │   │   ├── alpaca_gpt4_sharegpt_en_clean2.json\n│   │   │   ├── lima.json\n│   │   │   ├── oasst2.json\n│   │   ├── llava_v1_5_mix665k.json\n│   ├── llmga-imgs\n│   │   ├── COCO\n│   │   ├── LAION\n│   │   ├── JourneyDB\n│   ├── llava_pretrain\n│   │   ├──images\n│   ├── llava-imgs\n│   │   ├── coco\n│   │   │   ├── train2017\n│   │   ├── gqa\n│   │   │   ├── images\n│   │   ├── ocr_vqa\n│   │   │   ├── images\n│   │   ├── textvqa\n│   │   │   ├── train_images\n│   │   ├── vg\n│   │   │   ├── VG_100K\n│   │   │   ├── VG_100K_2\n```\n\n\n\n## Train\n\nLLMGA is trained on 8 A100 GPUs with 80GB memory. To train on fewer GPUs, you can reduce the `per_device_train_batch_size` and increase the `gradient_accumulation_steps` accordingly. Always keep the global batch size the same: `per_device_train_batch_size` x `gradient_accumulation_steps` x `num_gpus`.\n\nPlease make sure you download and organize the data following [Preparation](#Preparation) before training. Here, we just take training llmga vicuna 7b as an example. For more model training scripts, please check the **./scripts** folder.  \n\n### Pretrain\n\n```bash\nbash scripts/pretrain_vicuna_7b.sh\n```\n\n### First Stage Training\n\n```bash\nbash scripts/train_llmga_s1_7b_vicuna.sh\n```\n\n### Second Stage Training\n\ntrain LLMGA based on SD1.5-T2I\n```bash\nbash scripts/train_llmga_s2_sd15_t2i.sh\n```\n\ntrain LLMGA based on SD1.5-Inpainting\n```bash\nbash scripts/train_llmga_s2_sd15_inpaint.sh\n```\n\n## Inference\n\n### CLI Inference\n\nUse LLMGA without the need of Gradio interface. It also supports multiple GPUs, 4-bit and 8-bit quantized inference. With 4-bit quantization.\nHere, we just give some examples for T2I, inpainting and instruction-based editing. For more model inference scripts, please check the **./scripts** folder. \n\nFor **T2I** generation task.\n```bash\nbash scripts/test-llmga-sdxl-t2i.sh\n```\n\nFor **inpainting or outpainting** task.\n```bash\nbash scripts/test-llmga-sd15-inpainting.sh\n```\n\nFor **instruction based editing** task.\n```bash\nbash scripts/test-llmga-sd15-editing.sh\n```\n\n### Gradio Inference\n```bash\nbash scripts/run_gradio_t2i.sh\n```\n\n\n\n\n\n\n## Citation\nIf you find this repo useful for your research, please consider citing the paper\n```\n@article{xia2023llmga,\n  title={LLMGA: Multimodal Large Language Model based Generation Assistant},\n  author={Xia, Bin and Wang, Shiyin, and Tao, Yingfan and Wang, Yitong and Jia, Jiaya},\n  journal={ECCV},\n  year={2024}\n}\n```\n\n## Acknowledgement\nWe would like to thank the following repos for their great work:\n\n- This work utilizes MLLM from [LLaVA](https://github.com/haotian-liu/LLaVA).\n- This work utilizes Stable Diffusion from [diffusers](https://github.com/huggingface/diffusers).\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdvlab-research%2Fllmga","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdvlab-research%2Fllmga","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdvlab-research%2Fllmga/lists"}