Awesome_Multimodel_LLM

Awesome_Multimodel is a curated GitHub repository that provides a comprehensive collection of resources for Multimodal Large Language Models (MLLM). It covers datasets, tuning techniques, in-context learning, visual reasoning, foundational models, and more. Stay updated with the latest advancement.
https://github.com/Atomic-man007/Awesome_Multimodel_LLM

Last synced: 1 day ago
JSON representation

Compressing memories with vectors or data structures
- 2023/07
- 2023/05
- 2023/05 - lora)]
Courses about LLM
- DeepLearning.AI
- Princeton
- OpenBMB
- Stanford
- Stanford Webinar
- 李沐
- 陳縕儂
- 李沐
- 李沐 - 7UM8iUTj3qKqdhbQULP5I&index=18)
- Aston Zhang - 7UM8iUTj3qKqdhbQULP5I&index=29)
- Stanford
- Stanford
- MIT
Datasets of In-Context Learning
- MIMIC-IT: Multi-Modal In-Context Instruction Tuning - context instruction dataset|
- MIMIC-IT: Multi-Modal In-Context Instruction Tuning - context instruction dataset|
Datasets of Multimodal Chain-of-Thought
- EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought - scale embodied planning dataset |
- Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering - download-the-dataset) | Large-scale multi-choice dataset, featuring multimodal science questions and diverse domains |
- EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought - scale embodied planning dataset |
- Let’s Think Frame by Frame: Evaluating Video Chain of Thought with Video Infilling and Prediction - time dataset that can be used to evaluate VideoCOT |
Datasets of Multimodal Instruction Tuning
- X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages - LLM) | Chinese multimodal instruction dataset |
- GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction - related instruction datasets |
- ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst - chatbridge.github.io/) | Multimodal instruction tuning dataset covering 16 multimodal tasks |
- DetGPT: Detect What You Need via Reasoning - tuning dataset with 5000 images and around 30000 query-answer pairs|
- VideoChat: Chat-Centric Video Understanding - centric multimodal instruction dataset |
- mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality - PLUG/mPLUG-Owl/tree/main/OwlEval) | Dataset for evaluation on multiple capabilities |
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models - CAIR/cc_sbu_align) | Multimodal aligned dataset for improving model's usability and generation's fluency |
- Visual Instruction Tuning - Instruct-150K) | Multimodal instruction-following data generated by GPT|
- MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning - | The first multimodal instruction tuning benchmark dataset |
- X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages - LLM) | Chinese multimodal instruction dataset |
- GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction - related instruction datasets |
- DetGPT: Detect What You Need via Reasoning - tuning dataset with 5000 images and around 30000 query-answer pairs|
- Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models - oryx/Video-ChatGPT#video-instruction-dataset-open_file_folder) | 100K high-quality video instruction dataset |
- M<sup>3</sup>IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning - scale, broad-coverage multimodal instruction tuning dataset |
- ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst - chatbridge.github.io/) | Multimodal instruction tuning dataset covering 16 multimodal tasks |
- PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering - zhang.github.io/PMC-VQA/) | Large-scale medical visual question-answering dataset |
- VideoChat: Chat-Centric Video Understanding - centric multimodal instruction dataset |
- mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality - PLUG/mPLUG-Owl/tree/main/OwlEval) | Dataset for evaluation on multiple capabilities |
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models - CAIR/cc_sbu_align) | Multimodal aligned dataset for improving model's usability and generation's fluency |
- Visual Instruction Tuning - Instruct-150K) | Multimodal instruction-following data generated by GPT|
- LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day - Med#llava-med-dataset) | A large-scale, broad-coverage biomedical instruction-following dataset |
Datasets of Pre-Training for Alignment
Deep understanding
- 2022/08
- 2021/10
- 2022/08
- 2023/06
Foundation Models
- **PaLM-E: An Embodied Multimodal Language Model** - 03-06 | - | [Demo](https://palm-e.github.io/#demo) |
- **GPT-4 Technical Report** - 03-15 | - | - |
- Star - 02-27 | [Github](https://github.com/microsoft/unilm) | - |
High-quality generation
- 2023/10 - icler/PCA-EVAL)]
- 2023/06
- 2023/04
- 2023/06
- 2023/04
LLM-Aided Visual Reasoning
- Star - 05-24 | [Github](https://github.com/weixi-feng/LayoutGPT) | - |
- Star - 2 Answers: Automatic Questioning Towards Enriched Visual Descriptions**](https://arxiv.org/pdf/2303.06594.pdf) <br> | arXiv | 2023-03-12 | [Github](https://github.com/Vision-CAIR/ChatCaptioner) | Local Demo |
LLM Leaderboards
- 2017
  - [Source
  - [Source
  - [Source
  - [Source
  - [Source
  - [Source
  - [Source
  - [Source
  - [Source
  - [Source
  - [Source
  - [Source
  - [Source
  - [Source
  - [Source
  - [Source
  - [Source
  - [Source
  - [Source
  - [Source
  - [Source
  - [Source
  - [Source
  - [Source
  - [Source
  - [Source
LLM Training Frameworks
- FairScale - FairScale is a PyTorch extension library for high performance and large scale training.
- Alpa - Alpa is a system for training and serving large-scale neural networks.
- Megatron-LM - Ongoing research training transformer models at scale.
- Colossal-AI - Making large AI models cheaper, faster, and more accessible.
- BMTrain - Efficient Training for Big Models.
- Mesh Tensorflow - Mesh TensorFlow: Model Parallelism Made Easier.
- maxtext - A simple, performant and scalable Jax LLM!
- GPT-NeoX - An implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library.
- DeepSpeed - DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
- maxtext - A simple, performant and scalable Jax LLM!
Memory retrieval
- 2023/05 - siliconfriend)]
- 2023/04 - research/generative_agents)]
- 2023/05 - cn/RecurrentGPT)]
- 2023/06
- 2023/08
- 2023/08
- 2023/05 - cn/RecurrentGPT)]

Programming Languages

Python 48 Jupyter Notebook 8 TypeScript 4 Rust 2 JavaScript 2 C 1 C# 1 MDX 1 HTML 1 C++ 1

Categories

Practical Guide for NLP Tasks 142 RLHFdataset 130 Datasets of Pre-Training for Alignment 100 Open Source LLM 56 Milestone Papers 54 LLM Leaderboards 26 Tutorials about LLM 24 Prompting libraries & tools 23 Datasets of Multimodal Instruction Tuning 21 Other useful resources 15 Courses about LLM 13 Raising the length limit of Transformers 12 Pretraining data 11 Tools for deploying LLM 10 LLM Training Frameworks 10 Multimodal Chain-of-Thought 8 Memory retrieval 7 Trending LLM Projects 7 Multimodal Instruction Tuning 7 High-quality generation 5 Datasets of Multimodal Chain-of-Thought 4 Deep understanding 4 Multimodal In-Context Learning 4 Others 4 Compressing memories with vectors or data structures 3 Star History 3 Foundation Models 3 LLM-Aided Visual Reasoning 2 Datasets of In-Context Learning 2 Practical Guides for Prompting (Helpful) 2

Sub Categories

Efficiency 174 2017 82 Specific tasks 13 2023 7 Abilities with Scaling 6 Knowledge-intensive tasks 5 Generation tasks 5 2022 3 Test data/user data 3 Traditional NLU tasks 3 Finetuning data 3 2021 2 Real-World ''Tasks'' 1

Keywords

llm 16 large-language-models 13 deep-learning 12 chatgpt 12 language-model 11 gpt 8 machine-learning 8 llama 8 pytorch 7 transformers 7 ai 6 prompt-engineering 6 gpt-3 6 generative-ai 5 openai 5 nlp 5 chinese 4 artificial-intelligence 4 transformer 4 inference 4 python 3 chatbot 3 gpu 3 natural-language-processing 3 prompt-tuning 2 gpt-2 2 mistral 2 knowlm 2 instruction-tuning 2 ollama 2 agents 2 embeddings 2 evaluation 2 huggingface 2 rwkv 2 rnn 2 instruction-following 2 rlhf 2 aigc 2 bloom 2 foundation-models 2 fine-tuning 2 framework 2 gpt4 2 lora 2 data-parallelism 2 model-parallelism 2 stable-diffusion 2 pipeline-parallelism 2 deepseek 2

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

Awesome_Multimodel_LLM

Compressing memories with vectors or data structures

Courses about LLM

Datasets of In-Context Learning

Datasets of Multimodal Chain-of-Thought

Datasets of Multimodal Instruction Tuning

Datasets of Pre-Training for Alignment

Deep understanding

Foundation Models

High-quality generation

LLM-Aided Visual Reasoning

LLM Leaderboards

2017

LLM Training Frameworks

Memory retrieval