{"id":13645156,"url":"https://github.com/open-mmlab/multimodal-gpt","last_synced_at":"2025-05-15T23:04:26.751Z","repository":{"id":157033354,"uuid":"632871496","full_name":"open-mmlab/Multimodal-GPT","owner":"open-mmlab","description":"Multimodal-GPT","archived":false,"fork":false,"pushed_at":"2023-06-04T01:42:37.000Z","size":112,"stargazers_count":1498,"open_issues_count":22,"forks_count":131,"subscribers_count":13,"default_branch":"main","last_synced_at":"2025-05-11T00:32:12.713Z","etag":null,"topics":["flamingo","gpt","gpt-4","llama","multimodal","transformer","vision-and-language"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/open-mmlab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-26T09:54:07.000Z","updated_at":"2025-04-28T08:35:30.000Z","dependencies_parsed_at":null,"dependency_job_id":"e7be7a92-8289-49ad-87e2-3d7a88e73a0b","html_url":"https://github.com/open-mmlab/Multimodal-GPT","commit_stats":{"total_commits":22,"total_committers":12,"mean_commits":"1.8333333333333333","dds":0.7272727272727273,"last_synced_commit":"9c73e47ad6c339e828a44f164d1a2c5bff904747"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-mmlab%2FMultimodal-GPT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-mmlab%2FMultimodal-GPT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-mmlab%2FMultimodal-GPT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-mmlab%2FMultimodal-GPT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/open-mmlab","download_url":"https://codeload.github.com/open-mmlab/Multimodal-GPT/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254436944,"owners_count":22070946,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["flamingo","gpt","gpt-4","llama","multimodal","transformer","vision-and-language"],"created_at":"2024-08-02T01:02:29.877Z","updated_at":"2025-05-15T23:04:26.710Z","avatar_url":"https://github.com/open-mmlab.png","language":"Python","funding_links":[],"categories":["Langchain"],"sub_categories":[],"readme":"# 🤖 Multi-modal GPT\n\nTrain a multi-modal chatbot with visual and language instructions!\n\nBased on the open-source multi-modal model [OpenFlamingo](https://github.com/mlfoundations/open_flamingo), we create various **visual instruction** data with open datasets, including VQA, Image Captioning, Visual Reasoning, Text OCR, and Visual Dialogue. Additionally, we also train the language model component of OpenFlamingo using only **language-only instruction** data.\n\nThe **joint training** of visual and language instructions effectively improves the performance of the model! For more details please refer to our [technical report](https://arxiv.org/abs/2305.04790).\n\nWelcome to join us!\n\n\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\n\nEnglish | [简体中文](README_zh-CN.md)\n\n\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"https://openmmlab.medium.com/\" style=\"text-decoration:none;\"\u003e\n    \u003cimg src=\"https://user-images.githubusercontent.com/25839884/219255827-67c1a27f-f8c5-46a9-811d-5e57448c61d1.png\" width=\"3%\" alt=\"\" /\u003e\u003c/a\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/25839884/218346358-56cc8e2f-a2b8-487f-9088-32480cceabcf.png\" width=\"3%\" alt=\"\" /\u003e\n  \u003ca href=\"https://discord.com/channels/1037617289144569886/1046608014234370059\" style=\"text-decoration:none;\"\u003e\n    \u003cimg src=\"https://user-images.githubusercontent.com/25839884/218347213-c080267f-cbb6-443e-8532-8e1ed9a58ea9.png\" width=\"3%\" alt=\"\" /\u003e\u003c/a\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/25839884/218346358-56cc8e2f-a2b8-487f-9088-32480cceabcf.png\" width=\"3%\" alt=\"\" /\u003e\n  \u003ca href=\"https://twitter.com/OpenMMLab\" style=\"text-decoration:none;\"\u003e\n    \u003cimg src=\"https://user-images.githubusercontent.com/25839884/218346637-d30c8a0f-3eba-4699-8131-512fb06d46db.png\" width=\"3%\" alt=\"\" /\u003e\u003c/a\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/25839884/218346358-56cc8e2f-a2b8-487f-9088-32480cceabcf.png\" width=\"3%\" alt=\"\" /\u003e\n  \u003ca href=\"https://www.youtube.com/openmmlab\" style=\"text-decoration:none;\"\u003e\n    \u003cimg src=\"https://user-images.githubusercontent.com/25839884/218346691-ceb2116a-465a-40af-8424-9f30d2348ca9.png\" width=\"3%\" alt=\"\" /\u003e\u003c/a\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/25839884/218346358-56cc8e2f-a2b8-487f-9088-32480cceabcf.png\" width=\"3%\" alt=\"\" /\u003e\n  \u003ca href=\"https://space.bilibili.com/1293512903\" style=\"text-decoration:none;\"\u003e\n    \u003cimg src=\"https://user-images.githubusercontent.com/25839884/219026751-d7d14cce-a7c9-4e82-9942-8375fca65b99.png\" width=\"3%\" alt=\"\" /\u003e\u003c/a\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/25839884/218346358-56cc8e2f-a2b8-487f-9088-32480cceabcf.png\" width=\"3%\" alt=\"\" /\u003e\n  \u003ca href=\"https://www.zhihu.com/people/openmmlab\" style=\"text-decoration:none;\"\u003e\n    \u003cimg src=\"https://user-images.githubusercontent.com/25839884/219026120-ba71e48b-6e94-4bd4-b4e9-b7d175b5e362.png\" width=\"3%\" alt=\"\" /\u003e\u003c/a\u003e\n\u003c/div\u003e\n\n## Features\n\n- Support various vision and language instruction data\n- Parameter efficient fine-tuning with LoRA\n- Tuning vision and language at the same time, complement each other\n\n\n## Installation\n\nTo install the package in an existing environment, run\n\n```bash\ngit clone https://github.com/open-mmlab/Multimodal-GPT.git\ncd Multimodal-GPT\npip install -r requirements.txt\npip install -v -e .\n```\n\nor create a new conda environment\n\n```bash\nconda env create -f environment.yml\n```\n\n\n## Launch Demo Locally\n\n1. Download the pre-trained weights.\n\n    Use [this script](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py) for converting LLaMA weights to Hugging Face format.\n\n    Download the OpenFlamingo pre-trained model from [openflamingo/OpenFlamingo-9B](https://huggingface.co/openflamingo/OpenFlamingo-9B).\n\n    Download our LoRA Weight from [here](https://download.openmmlab.com/mmgpt/v0/mmgpt-lora-v0-release.pt).\n\n    Then place these models in `checkpoints` folders like this:\n\n    ```\n    checkpoints\n    ├── llama-7b_hf\n    │   ├── config.json\n    │   ├── pytorch_model-00001-of-00002.bin\n    │   ├── ......\n    │   └── tokenizer.model\n    ├── OpenFlamingo-9B\n    │   └──checkpoint.pt\n    ├──mmgpt-lora-v0-release.pt\n\n2. launch the gradio demo\n\n    ```bash\n    python app.py\n    ```\n\n## Examples\n\n### Recipe:\n![image4](https://user-images.githubusercontent.com/12907710/234554562-8f3be88f-d563-47ba-97d9-ade8d47c46b0.png)\n\n### Travel plan:\n![image3](https://user-images.githubusercontent.com/12907710/234523464-80c4e3f0-f99f-4498-96ef-dc43ef89c64b.png)\n\n### Movie:\n![image2](https://user-images.githubusercontent.com/12907710/234523468-e11905a6-491f-4b87-934f-90da7d14d1c3.png)\n\n### Famous person:\n![image](https://user-images.githubusercontent.com/12907710/234523475-fd91f979-a344-4228-813f-6b55a1bc250f.png)\n\n\n## Fine-tuning\n\n### Prepare datasets\n\n1. [A-OKVQA](https://allenai.org/project/a-okvqa/home)\n\n    Download annotation from [this link](https://prior-datasets.s3.us-east-2.amazonaws.com/aokvqa/aokvqa_v1p0.tar.gz) and unzip to `data/aokvqa/annotations`.\n\n    It also requires images from coco dataset which can be downloaded from [here](https://cocodataset.org/#home). \n\n2. [COCO Caption](https://cs.stanford.edu/people/karpathy/deepimagesent/)\n\n    Download from [this link](https://cs.stanford.edu/people/karpathy/deepimagesent/coco.zip) and unzip to `data/coco`.\n\n    It also requires images from coco dataset which can be downloaded from [here](https://cocodataset.org/#home).\n\n3. [OCR VQA](https://ocr-vqa.github.io/)\n\n    Download from [this link](https://drive.google.com/drive/folders/1_GYPY5UkUy7HIcR0zq3ZCFgeZN7BAfm_?usp=sharing) and place in `data/OCR_VQA/`.\n\n4. [LlaVA](https://llava-vl.github.io/)\n\n    Download from [liuhaotian/LLaVA-Instruct-150K](https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K) and place in `data/llava/`.\n\n    It also requires images from coco dataset which can be downloaded from [here](https://cocodataset.org/#home).\n\n5. [Mini-GPT4](https://minigpt-4.github.io/)\n\n    Download from [Vision-CAIR/cc_sbu_align](https://huggingface.co/datasets/Vision-CAIR/cc_sbu_align) and place in `data/cc_sbu_align/`.\n\n6. [Dolly 15k](https://www.databricks.com/blog/2023/03/24/hello-dolly-democratizing-magic-chatgpt-open-models.html)\n\n    Download from [databricks/databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k) and place it in `data/dolly/databricks-dolly-15k.jsonl`.\n\n7. [Alpaca GPT4](https://github.com/Instruction-Tuning-with-GPT-4/GPT-4-LLM)\n\n    Download it from [this link](https://github.com/Instruction-Tuning-with-GPT-4/GPT-4-LLM/raw/main/data/alpaca_gpt4_data.json) and place it in `data/alpaca_gpt4/alpaca_gpt4_data.json`.\n\nYou can also customize the data path in the [configs/dataset_config.py](configs/dataset_config.py).\n\n8. [Baize](https://github.com/project-baize/baize-chatbot)\n\n    Download it from [this link](https://github.com/project-baize/baize-chatbot/blob/main/data/quora_chat_data.json) and place it in `data/baize/quora_chat_data.json`.\n\n\n## Start training\n\n```bash\ntorchrun --nproc_per_node=8 mmgpt/train/instruction_finetune.py \\\n  --lm_path checkpoints/llama-7b_hf \\\n  --tokenizer_path checkpoints/llama-7b_hf \\\n  --pretrained_path checkpoints/OpenFlamingo-9B/checkpoint.pt \\\n  --run_name train-my-gpt4 \\\n  --learning_rate 1e-5 \\\n  --lr_scheduler cosine \\\n  --batch_size 1 \\ \n  --tuning_config configs/lora_config.py \\\n  --dataset_config configs/dataset_config.py \\\n  --report_to_wandb\n```\n\n\n## Acknowledgements\n\n- [OpenFlamingo](https://github.com/mlfoundations/open_flamingo)\n- [LAVIS](https://github.com/salesforce/LAVIS)\n- [Stanford Alpaca](https://github.com/tatsu-lab/stanford_alpaca)\n- [MiniGPT-4](https://github.com/Vision-CAIR/MiniGPT-4)\n- [LLaVA](https://github.com/haotian-liu/LLaVA/tree/main)\n- [Instruction Tuning with GPT-4](https://github.com/Instruction-Tuning-with-GPT-4/GPT-4-LLM)\n\nIf you find our project useful for your research and applications, please cite using this BibTeX:\n\n```bibtex\n@misc{gong2023multimodalgpt,\n      title={MultiModal-GPT: A Vision and Language Model for Dialogue with Humans}, \n      author={Tao Gong and Chengqi Lyu and Shilong Zhang and Yudong Wang and Miao Zheng and Qian Zhao and Kuikun Liu and Wenwei Zhang and Ping Luo and Kai Chen},\n      year={2023},\n      eprint={2305.04790},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopen-mmlab%2Fmultimodal-gpt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopen-mmlab%2Fmultimodal-gpt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopen-mmlab%2Fmultimodal-gpt/lists"}