Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/showlab/awesome-unified-multimodal-models
📖 This is a repository for organizing papers, codes and other resources related to unified multimodal models.
https://github.com/showlab/awesome-unified-multimodal-models
List: awesome-unified-multimodal-models
Last synced: about 1 month ago
JSON representation
📖 This is a repository for organizing papers, codes and other resources related to unified multimodal models.
- Host: GitHub
- URL: https://github.com/showlab/awesome-unified-multimodal-models
- Owner: showlab
- Created: 2024-08-27T13:06:04.000Z (4 months ago)
- Default Branch: main
- Last Pushed: 2024-11-03T17:32:23.000Z (about 2 months ago)
- Last Synced: 2024-11-03T18:26:33.551Z (about 2 months ago)
- Homepage:
- Size: 611 KB
- Stars: 199
- Watchers: 16
- Forks: 6
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- ultimate-awesome - awesome-unified-multimodal-models - 📖 This is a repository for organizing papers, codes and other resources related to unified multimodal models. (Other Lists / PowerShell Lists)
README
# Awesome Unified Multimodal Models [](https://github.com/sindresorhus/awesome)
This is a repository for organizing papers, codes and other resources related to unified multimodal models.
#### :thinking: What are unified multimodal models?
Traditional multimodal models can be broadly categorized into two types: **multimodal understanding** and **multimodal generation**.
Unified multimodal models aim to integrate these two tasks within a single framework.
Such models are also referred to as Any-to-Any generation in the community.
These models operate on the principle of multimodal input and multimodal output, enabling them to process and generate content across various modalities seamlessly.
#### :high_brightness: This project is still on-going, pull requests are welcomed!!
If you have any suggestions (missing papers, new papers, or typos), please feel free to edit and pull a request. Just letting us know the title of papers can also be a great contribution to us. You can do this by open issue or contact us directly via email.
#### :star: If you find this repo useful, please star it!!!
## Table of Contents
- [Open-source Toolboxes and Foundation Models](#open-source-toolboxes-and-foundation-models)
- [Evaluation Benchmarks and Metrics](#evaluation-benchmarks-and-metrics)
- [Single Model ](#single-model)
- [Multi Experts](#multi-experts)
- [Tokenizer](#tokenizers)
### Unified Multimodal Understanding and Generation
+ [MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understanding](https://arxiv.org/pdf/2410.21747) (Oct. 2024, arXiv)
[](https://arxiv.org/pdf/2410.21747)
+ [Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation](https://arxiv.org/abs/2410.13848) (Oct. 2024, arXiv)
[](https://arxiv.org/abs/2409.18869)
[](https://github.com/deepseek-ai/Janus)
+ [MMAR: Towards Lossless Multi-Modal Auto-Regressive Probabilistic Modeling](https://arxiv.org/abs/2410.10798) (Oct. 2024, arXiv)
[](https://arxiv.org/abs/2410.10798)
+ [Emu3: Next-Token Prediction is All You Need](https://arxiv.org/abs/2409.18869) (Sep. 2024, arXiv)
[](https://arxiv.org/abs/2409.18869)
[](https://github.com/baaivision/Emu3)
+ [MIO: A Foundation Model on Multimodal Tokens](https://arxiv.org/abs/2409.17692) (Sep. 2024, arXiv)
[](https://arxiv.org/abs/2409.17692)
+ [MonoFormer: One Transformer for Both Diffusion and Autoregression](https://arxiv.org/abs/2409.16280) (Sep. 2024, arXiv)
[](https://arxiv.org/abs/2409.16280)
[](https://github.com/MonoFormer/MonoFormer)
+ [VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation](https://arxiv.org/abs/2409.04429) (Sep. 2024, arXiv)
[](https://arxiv.org/abs/2409.04429)
+ [Show-o: One Single Transformer to Unify Multimodal Understanding and Generation](https://arxiv.org/abs/2408.12528) (Aug. 2024, arXiv)
[](https://arxiv.org/abs/2408.12528)
[](https://github.com/showlab/Show-o)
+ [Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model](https://www.arxiv.org/abs/2408.11039) (Aug. 2024, arXiv)
[](https://www.arxiv.org/abs/2408.11039)
+ [ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generation](https://arxiv.org/abs/2407.06135) (Jul. 2024, arXiv)
[](https://arxiv.org/abs/2407.06135)
[](https://github.com/GAIR-NLP/anole)
+ [X-VILA: Cross-Modality Alignment for Large Language Model](https://arxiv.org/abs/2405.19335) (May. 2024, arXiv)
[](https://arxiv.org/abs/2405.19335)
+ [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://arxiv.org/abs/2405.09818) (May 2024, arXiv)
[](https://arxiv.org/abs/2405.09818)
[](https://github.com/facebookresearch/chameleon)
+ [SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation](https://arxiv.org/abs/2404.14396) (Apr. 2024, arXiv)
[](https://arxiv.org/abs/2404.14396)
[](https://github.com/AILab-CVC/SEED-X)
+ [Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models](https://arxiv.org/abs/2403.18814) (Mar. 2024, arXiv)
[](https://arxiv.org/abs/2403.18814)
[](https://github.com/dvlab-research/MGM)
+ [AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling](https://arxiv.org/abs/2402.12226) (Feb. 2024, arXiv)
[](https://arxiv.org/abs/2402.12226)
[](https://github.com/OpenMOSS/AnyGPT)
+ [World Model on Million-Length Video And Language With Blockwise RingAttention](https://arxiv.org/abs/2402.08268) (Feb. 2024, arXiv)
[](https://arxiv.org/abs/2402.08268)
[](https://github.com/LargeWorldModel/LWM)
+ [Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization](https://arxiv.org/abs/2402.03161) (Feb. 2024, arXiv)
[](https://arxiv.org/abs/2402.03161)
[](https://github.com/jy0205/LaVIT)
+ [MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer](https://arxiv.org/abs/2401.10208) (Jan. 2024, arXiv)
[](https://arxiv.org/abs/2401.10208)
[](https://github.com/OpenGVLab/MM-Interleaved)
+ [Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action](https://arxiv.org/abs/2312.17172) (Dec. 2023, arXiv)
[](https://arxiv.org/abs/2312.17172)
[](https://github.com/allenai/unified-io-2)
+ [Emu2: Generative Multimodal Models are In-Context Learners](https://arxiv.org/abs/2312.13286) (Jul. 2023, CVPR)
[](https://arxiv.org/abs/2312.13286)
[](https://github.com/baaivision/Emu)
+ [Gemini: A Family of Highly Capable Multimodal Models](https://arxiv.org/abs/2312.11805) (Dec. 2023, arXiv)
[](https://arxiv.org/abs/2312.11805)
+ [VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generation](https://arxiv.org/abs/2312.09251) (Dec. 2023, arXiv)
[](https://arxiv.org/abs/2312.09251)
[](https://github.com/AILab-CVC/VL-GPT)
+ [DreamLLM: Synergistic Multimodal Comprehension and Creation](https://arxiv.org/abs/2309.11499) (Dec. 2023, ICLR)
[](https://arxiv.org/abs/2309.11499)
[](https://github.com/RunpeiDong/DreamLLM)
+ [Making LLaMA SEE and Draw with SEED Tokenizer](https://arxiv.org/abs/2310.01218) (Oct. 2023, ICLR)
[](https://arxiv.org/abs/2310.01218)
[](https://github.com/AILab-CVC/SEED/)
+ [NExT-GPT: Any-to-Any Multimodal LLM](https://arxiv.org/abs/2309.05519) (Sep. 2023, ICML)
[](https://arxiv.org/abs/2309.05519)
[](https://github.com/NExT-GPT/NExT-GPT)
+ [LaVIT: Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization](https://arxiv.org/abs/2309.04669) (Sep. 2023, ICLR)
[](https://arxiv.org/abs/2309.04669)
[](https://github.com/jy0205/LaVIT)
+ [Planting a SEED of Vision in Large Language Model](https://arxiv.org/abs/2307.08041) (Jul. 2023, arXiv)
[](https://arxiv.org/abs/2307.08041)
[](https://github.com/AILab-CVC/SEED/tree/v1)
+ [Emu: Generative Pretraining in Multimodality](https://arxiv.org/abs/2307.05222) (Jul. 2023, ICLR)
[](https://arxiv.org/abs/2307.05222)
[](https://github.com/baaivision/Emu)
+ [CoDi: Any-to-Any Generation via Composable Diffusion](https://arxiv.org/abs/2305.11846) (May. 2023, NeurIPS)
[](https://arxiv.org/abs/2305.11846)
[](https://github.com/microsoft/i-Code/tree/main/i-Code-V3)
+ [Multimodal unified attention networks for vision-and-language interactions](https://arxiv.org/abs/1908.04107) (Aug. 2019)
[](https://arxiv.org/abs/1908.04107)
+ [UniMuMo: Unified Text, Music, and Motion Generation](https://arxiv.org/abs/2410.04534) (Oct. 2024, arXiv)
[](https://arxiv.org/abs/2410.04534)
[](https://github.com/hanyangclarence/UniMuMo)
[](https://hanyangclarence.github.io/unimumo_demo/)
+ [MedViLaM: A multimodal large language model with advanced generalizability and explainability for medical data understanding and generation](https://arxiv.org/pdf/2409.19684) (Oct. 2024, arXiv)
[](https://arxiv.org/pdf/2409.19684)
### Multi Experts
+ [TaxaBind: A Unified Embedding Space for Ecological Applications](https://arxiv.org/pdf/2411.00683) (Nov. 2024, arXiv)
[](https://arxiv.org/pdf/2411.00683)
[](https://github.com/mvrl/TaxaBind)
[](https://vishu26.github.io/taxabind/index.html)
### Tokenizer
+ [Cosmos Tokenizer: A suite of image and video neural tokenizers](https://developer.nvidia.com/blog/state-of-the-art-multimodal-generative-ai-model-development-with-nvidia-nemo/) (Nov. 2024, arXiv)
[](https://developer.nvidia.com/blog/state-of-the-art-multimodal-generative-ai-model-development-with-nvidia-nemo/)
[](https://github.com/NVIDIA/Cosmos-Tokenizer)
[](https://research.nvidia.com/labs/dir/cosmos-tokenizer/)
## Acknowledgements
This template is provided by [Awesome-Video-Diffusion](https://github.com/showlab/Awesome-Video-Diffusion) and [Awesome-MLLM-Hallucination](https://github.com/showlab/Awesome-MLLM-Hallucination).