Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/SkalskiP/awesome-foundation-and-multimodal-models
ποΈ + π¬ + π§ = π€ Curated list of top foundation and multimodal models! [Paper + Code + Examples + Tutorials]
https://github.com/SkalskiP/awesome-foundation-and-multimodal-models
List: awesome-foundation-and-multimodal-models
blip clip computer-vision foundational-models grounding-dino image-captioning llava multimodal nlp open-vocabulary-detection open-vocabulary-segmentation segment-anything zero-shot-detection
Last synced: about 1 month ago
JSON representation
ποΈ + π¬ + π§ = π€ Curated list of top foundation and multimodal models! [Paper + Code + Examples + Tutorials]
- Host: GitHub
- URL: https://github.com/SkalskiP/awesome-foundation-and-multimodal-models
- Owner: SkalskiP
- Created: 2023-10-08T13:30:23.000Z (about 1 year ago)
- Default Branch: master
- Last Pushed: 2024-02-29T14:14:56.000Z (9 months ago)
- Last Synced: 2024-05-22T01:10:32.744Z (6 months ago)
- Topics: blip, clip, computer-vision, foundational-models, grounding-dino, image-captioning, llava, multimodal, nlp, open-vocabulary-detection, open-vocabulary-segmentation, segment-anything, zero-shot-detection
- Language: Python
- Homepage:
- Size: 58.6 KB
- Stars: 518
- Watchers: 22
- Forks: 39
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesome-foundation-model-ros - Awesome Foundation and Multimodal Models - Curated list of top foundation and multimodal models (Research-Grade Frameworks)
- awesome-osml-for-devs - Awesome Open-source Foundation and Multimodal Models - shot capabilities and the models that can take in two different modalities (e.g. a model that can take in image and text and output text). (Multimodal Models / Tools)
- awesome-osml-for-devs - Awesome Open-source Foundation and Multimodal Models - shot capabilities and the models that can take in two different modalities (e.g. a model that can take in image and text and output text). (Multimodal Models / Tools)
README
awesome foundation and multimodal models
## ποΈ + π¬ + π§ = π€
**foundation model** - a pre-trained machine learning model that serves as a base for a wide range of downstream tasks. It captures general knowledge from a large dataset and can be fine-tuned to perform specific tasks more effectively.
**multimodal model** - a model that can process multiple modalities (e.g. text, image,
video, audio, etc.) at the same time.
## π€ models
### YOLO-World: Real-Time Open-Vocabulary Object Detection
[](https://arxiv.org/abs/2401.17270) [](https://github.com/AILab-CVC/YOLO-World) [](https://youtu.be/X7gKBGVz4vs) [](https://huggingface.co/spaces/SkalskiP/YOLO-World) [](https://colab.research.google.com/github/roboflow-ai/notebooks/blob/main/notebooks/zero-shot-object-detection-with-yolo-world.ipynb)
Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, Ying Shan
- **Date:** 2024-01-30
- **Modalities:** ποΈ + π¬
- **Tasks:** Zero-Shot Object Detection
### Depth Anything
[](https://arxiv.org/abs/2401.10891) [](https://github.com/LiheYoung/Depth-Anything) [](https://huggingface.co/spaces/LiheYoung/Depth-Anything) [](https://huggingface.co/LiheYoung/depth_anything_vitl14) [](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/Depth%20Anything/Predicting_depth_in_an_image_with_Depth_Anything.ipynb)
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, Hengshuang Zhao
- **Date:** 2024-01-19
- **Modalities:** π
- **Tasks:** Depth Estimation
### EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything
[](https://arxiv.org/abs/2312.00863) [](https://github.com/yformer/EfficientSAM) [](https://huggingface.co/spaces/SkalskiP/EfficientSAM) [](https://huggingface.co/merve/EfficientSAM)
Yunyang Xiong, Bala Varadarajan, Lemeng Wu, Xiaoyu Xiang, Fanyi Xiao, Chenchen Zhu, Xiaoliang Dai, Dilin Wang, Fei Sun, Forrest Iandola, Raghuraman Krishnamoorthi, Vikas Chandra
- **Date:** 2023-12-01
- **Modalities:** ποΈ
- **Tasks:** Zero-Shot Object Segmentation
### Qwen-VL-Plus / Max
[](https://arxiv.org/abs/2308.12966) [](https://github.com/QwenLM/Qwen-VL#qwen-vl-plus) [](https://huggingface.co/spaces/Qwen/Qwen-VL-Plus) [](https://huggingface.co/Qwen/Qwen-VL)
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, Jingren Zhou
- **Date:** 2023-11-28
- **Modalities:** ποΈ + π¬
- **Tasks:** Image Captioning, VQA, Zero-Shot Object Detection
### CogVLM: Visual Expert for Pretrained Language Models
[](https://arxiv.org/abs/2311.03079) [](https://github.com/THUDM/CogVLM) [](https://huggingface.co/spaces/lykeven/CogVLM) [](https://huggingface.co/THUDM/CogVLM)
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, Jie Tang
- **Date:** 2023-11-06
- **Modalities:** ποΈ + π¬
- **Tasks:** Image Captioning, VQA
### Fuyu-8B: A Multimodal Architecture for AI Agents
[](https://huggingface.co/spaces/adept/fuyu-8b-demo) [](https://huggingface.co/adept/fuyu-8b)
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, SaΔnak TaΕΔ±rlar
- **Date:** 2023-10-17
- **Modalities:** ποΈ + π¬
- **Tasks:** Image Classification, Image Captioning, VQA, Find Text in Image
### Ferret: Refer and Ground Anything Anywhere at Any Granularity
[](https://arxiv.org/abs/2310.07704) [](https://github.com/apple/ml-ferret)
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, Yinfei Yang
- **Date:** 2023-10-11
- **Modalities:** ποΈ + π¬
- **Tasks:** Image Captioning, VQA, Phrase Grounding, Object Detection
### MetaCLIP: Demystifying CLIP Data
[](https://arxiv.org/abs/2309.16671) [](https://github.com/facebookresearch/MetaCLIP) [](https://huggingface.co/spaces/SkalskiP/SAM_and_MetaCLIP) [](https://huggingface.co/facebook/metaclip-b32-400m) [](https://colab.research.google.com/drive/1V0Rv1QQJkcolTjiwJuRsqWycROvYjOwg?usp=sharing)
Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, Christoph Feichtenhofer
- **Date:** 2023-09-28
- **Modalities:** ποΈ + π¬
- **Tasks:** Zero-Shot Classification
### Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
[](https://arxiv.org/abs/2308.12966) [](https://github.com/QwenLM/Qwen-VL) [](https://huggingface.co/spaces/Qwen/Qwen-VL-Max) [](https://huggingface.co/Qwen/Qwen-VL)
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, Jingren Zhou
- **Date:** 2023-09-24
- **Modalities:** ποΈ + π¬
- **Tasks:** Image Captioning, VQA
### SigLIP: Sigmoid Loss for Language Image Pre-Training
[](https://arxiv.org/abs/2303.15343) [](https://github.com/google-research/big_vision) [](https://huggingface.co/spaces/merve/compare_clip_siglip) [](https://huggingface.co/openai/clip-vit-base-patch16) [](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/SigLIP/Inference_with_(multilingual)_SigLIP%2C_a_better_CLIP_model.ipynb)
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, Lucas Beyer
- **Date:** 2023-08-27
- **Modalities:** ποΈπ¬
- **Tasks:** Zero-Shot Image Classification
### Nougat: Neural Optical Understanding for Academic Documents
[](https://arxiv.org/abs/2308.13418) [](https://github.com/facebookresearch/nougat) [](https://huggingface.co/spaces/hf-vision/nougat-transformers) [](https://huggingface.co/facebook/nougat-small) [](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/Nougat/Inference_with_Nougat_to_read_scientific_PDFs.ipynb)
Lukas Blecher, Guillem Cucurull, Thomas Scialom, Robert Stojnic
- **Date:** 2023-08-25
- **Modalities:** ποΈπ¬
- **Tasks:** Visual Question Answering
### AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining
[](https://arxiv.org/abs/2308.05734) [](https://github.com/haoheliu/AudioLDM2) [](https://huggingface.co/spaces/haoheliu/AudioLDM_48K_Text-to-HiFiAudio_Generation) [](https://huggingface.co/spaces/haoheliu/audioldm2-text2audio-text2music)
Haohe Liu, Qiao Tian, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Yuping Wang, Wenwu Wang, Yuxuan Wang, Mark D. Plumbley
- **Date:** 2023-08-10
- **Modalities:** π¬οΈ + π§
- **Tasks:** Text-to-Audio, Text-to-Speech
### OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models
[](https://arxiv.org/abs/2308.01390) [](https://github.com/mlfoundations/open_flamingo) [](https://huggingface.co/spaces/openflamingo/OpenFlamingo) [](https://huggingface.co/openflamingo/OpenFlamingo-9B-vitl-mpt7b)
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, Ludwig Schmidt
- **Date:** 2023-08-02
- **Modalities:** ποΈ + π¬
- **Tasks:** Image Classification, Image Captioning, VQA
### Kosmos-2: Grounding Multimodal Large Language Models to the World
[](https://arxiv.org/abs/2306.14824) [](https://github.com/microsoft/unilm/tree/master/kosmos-2) [](https://huggingface.co/spaces/ydshieh/Kosmos-2) [](https://huggingface.co/microsoft/kosmos-2-patch14-224)
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, Furu Wei
- **Date:** 2023-07-26
- **Modalities:** ποΈ + π¬
- **Tasks:** Image Captioning, VQA, Phrase Grounding
### OWLv2: Scaling Open-Vocabulary Object Detection
[](https://arxiv.org/abs/2306.09683) [](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit) [](https://huggingface.co/spaces/merve/owlv2) [](https://huggingface.co/google/owlv2-base-patch16-ensemble) [](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/OWLv2/Zero_and_one_shot_object_detection_with_OWLv2.ipynb)
Matthias Minderer, Alexey Gritsenko, Neil Houlsby
- **Date:** 2023-06-17
- **Modalities:** ποΈ
- **Tasks:** Zero-Shot Object Detection
### ImageBind: One Embedding Space To Bind Them All
[](https://arxiv.org/abs/2305.05665) [](https://github.com/facebookresearch/ImageBind) [](https://huggingface.co/spaces/JustinLin610/ImageBind_zeroshot_demo)
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, Ishan Misra
- **Date:** 2023-05-09
- **Modalities:** ποΈ + π¬ + π§
- **Tasks:**
### LLaVA: Large Language and Vision Assistant
[](https://arxiv.org/abs/2304.08485) [](https://github.com/haotian-liu/LLaVA) [](https://huggingface.co/spaces/badayvedat/LLaVA) [](https://huggingface.co/liuhaotian/llava-v1.6-34b)
Haotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae Lee
- **Date:** 2023-04-17
- **Modalities:** ποΈ + π¬
- **Tasks:** Vision Language Modeling
### Segment Anything
[](https://arxiv.org/abs/2304.02643) [](https://github.com/facebookresearch/segment-anything) [](https://youtu.be/D-D6ZmadzPE) [](https://huggingface.co/spaces/radames/candle-segment-anything-wasm) [](https://huggingface.co/facebook/sam-vit-base) [](https://colab.research.google.com/github/roboflow-ai/notebooks/blob/main/notebooks/how-to-segment-anything-with-sam.ipynb)
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr DollΓ‘r, Ross Girshick
- **Date:** 2023-04-05
- **Modalities:** ποΈ
- **Tasks:** Zero-Shot Object Segmentation
### Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection
[](https://arxiv.org/abs/2303.05499) [](https://github.com/IDEA-Research/GroundingDINO) [](https://youtu.be/cMa77r3YrDk) [](https://huggingface.co/spaces/ShilongLiu/Grounding_DINO_demo) [](https://huggingface.co/spaces/merve/Grounding_DINO_demo) [](https://colab.research.google.com/github/roboflow-ai/notebooks/blob/main/notebooks/zero-shot-object-detection-with-grounding-dino.ipynb)
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang
- **Date:** 2023-03-09
- **Modalities:** ποΈ + π¬
- **Tasks:** Phrase Grounding, Zero-Shot Object Detection
### BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
[](https://arxiv.org/abs/2301.12597) [](https://github.com/salesforce/LAVIS/tree/main/projects/blip2) [](https://huggingface.co/spaces/merve/BLIP2-with-transformers) [](https://huggingface.co/Salesforce/blip2-opt-6.7b) [](https://colab.research.google.com/github/salesforce/LAVIS/blob/main/examples/blip2_instructed_generation.ipynb)
Junnan Li, Dongxu Li, Silvio Savarese, Steven Hoi
- **Date:** 2023-01-30
- **Modalities:** ποΈ + π¬
- **Tasks:** Image Captioning, Visual Question Answering
### Whisper: Robust Speech Recognition via Large-Scale Weak Supervision
[](https://arxiv.org/abs/2212.04356) [](https://github.com/openai/whisper) [](https://huggingface.co/spaces/openai/whisper) [](https://huggingface.co/openai/whisper-large-v3) [](https://colab.research.google.com/github/openai/whisper/blob/master/notebooks/LibriSpeech.ipynb)
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, Ilya Sutskever
- **Date:** 2022-12-06
- **Modalities:** π¬οΈ + π§
- **Tasks:** Speech-to-Text
### OWL-ViT: Simple Open-Vocabulary Object Detection with Vision Transformers
[](https://arxiv.org/abs/2205.06230) [](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit) [](https://huggingface.co/spaces/adirik/OWL-ViT) [](https://huggingface.co/google/owlvit-base-patch32)
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, Neil Houlsby
- **Date:** 2022-05-12
- **Modalities:** ποΈ + π¬
- **Tasks:** Zero-Shot Object Detection
### CLIP: Learning Transferable Visual Models From Natural Language Supervision
[](https://arxiv.org/abs/2103.00020) [](https://github.com/openai/CLIP) [](https://huggingface.co/openai/clip-vit-large-patch14) [](https://colab.research.google.com/github/roboflow-ai/notebooks/blob/main/notebooks/how-to-use-openai-clip-classification.ipynb)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever
- **Date:** 2021-02-26
- **Modalities:** ποΈ + π¬
- **Tasks:** Zero-Shot Classification
## π¦Έ contribution
We would love your help in making this repository even better! If you know of an
amazing paper that isn't listed here, or if you have any suggestions for improvement,
feel free to open an [issue](https://github.com/SkalskiP/awesome-foundation-and-multimodal-models/issues)
or submit a [pull request](https://github.com/SkalskiP/awesome-foundation-and-multimodal-models/pulls).