https://github.com/haofanwang/awesome-vision-language-modeling

Recent Advances in Vision-Language Pre-training!
https://github.com/haofanwang/awesome-vision-language-modeling

List: awesome-vision-language-modeling

masked-image-modeling masked-language-models pre-training vision-language

Last synced: 4 months ago
JSON representation

Recent Advances in Vision-Language Pre-training!

• Host: GitHub
• URL: https://github.com/haofanwang/awesome-vision-language-modeling
• Owner: haofanwang
• Created: 2021-12-24T04:53:32.000Z (over 4 years ago)
• Default Branch: main
• Last Pushed: 2022-01-10T15:14:05.000Z (over 4 years ago)
• Last Synced: 2025-11-09T08:13:49.919Z (7 months ago)
• Topics: masked-image-modeling, masked-language-models, pre-training, vision-language
• Homepage:
• Size: 18.6 KB
• Stars: 29
• Watchers: 1
• Forks: 2
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# awesome-vision-language-pretraining [![Awesome](https://awesome.re/badge.svg)](https://awesome.re)

An up-to-date list of vision-language (VL) pre-training paper! Maintained by [Haofan Wang](https://haofanwang.github.io/) (haofanwang.ai@gmail.com).

## SSL+Pre-training

[Towards a Unified Foundation Model: Jointly Pre-Training Transformers on Unpaired Images and Text](https://arxiv.org/abs/2112.07074), Google.

[ERNIE-ViLG: Unified Generative Pre-training for Bidirectional Vision-Language Generation](https://arxiv.org/abs/2112.15283), Baidu.

*[FLAVA: A Foundational Language And Vision Alignment Model](https://arxiv.org/pdf/2112.04482.pdf), Meta AI.

[A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervision](https://arxiv.org/abs/2112.13884), Meta AI.

*[SLIP: Self-supervision meets Language-Image Pre-training](https://arxiv.org/abs/2112.12750), [[code](https://github.com/facebookresearch/SLIP)], Meta AI.

*[MLIM: Vision-and-Language Model Pre-training with Masked Language and Image Modeling](https://arxiv.org/abs/2109.12178), Amazon.

*[Data Efficient Masked Language Modeling for Vision and Language](https://arxiv.org/abs/2109.02040), [[code](https://github.com/yonatanbitton/data_efficient_masked_language_modeling_for_vision_and_language)], Ben Gurion University.

[VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://arxiv.org/abs/1908.08530), [[code](https://github.com/jackroos/VL-BERT)], MSRA.

[Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training](https://arxiv.org/pdf/1908.06066.pdf), [[code](https://github.com/microsoft/Unicoder)], MSRA.

[UNITER: UNiversal Image-TExt Representation Learning](https://arxiv.org/abs/1909.11740), [[code](https://github.com/ChenRocks/UNITER)], Microsoft.

[ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://arxiv.org/abs/1908.02265), [[code](https://github.com/facebookresearch/vilbert-multi-task)], Meta AI.

## Masked Image Modeling

[Masked Feature Prediction for Self-Supervised Visual Pre-Training](https://arxiv.org/abs/2112.09133), Meta AI.

[SimMIM: A Simple Framework for Masked Image Modeling](https://arxiv.org/abs/2111.09886), [[code](https://github.com/microsoft/SimMIM)], MSRA.

[PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers](https://arxiv.org/abs/2111.12710), [[code](https://github.com/microsoft/PeCo)], MSRA.

[Masked Autoencoders Are Scalable Vision Learners](https://arxiv.org/pdf/2111.06377.pdf), [[Unofficial code](https://github.com/pengzhiliang/MAE-pytorch)], Meta AI.

[iBOT: Image BERT Pre-Training with Online Tokenizer](https://arxiv.org/abs/2111.07832), [[code](https://github.com/bytedance/ibot)], ByteDance .

[BEiT: BERT Pre-Training of Image Transformers](https://arxiv.org/abs/2106.08254), [[code](https://github.com/microsoft/unilm)], MSRA.

## Masked Language Modeling

[BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805), [[code](https://github.com/google-research/bert)], Google.

## Others

[Align and Prompt: Video-and-Language Pre-training with Entity Prompts](https://arxiv.org/abs/2112.09583), Saleforce.

[FILIP: Fine-grained Interactive Language-Image Pre-Training](https://arxiv.org/pdf/2111.07783.pdf), [[Unofficial code](https://github.com/lucidrains/x-clip)], Huawei.

[LiT: Zero-Shot Transfer with Locked-image Text Tuning](https://arxiv.org/abs/2111.07991), [[Unofficial code](https://github.com/lucidrains/x-clip)], Google.

[Multimodal Few-Shot Learning with Frozen Language Models](https://proceedings.neurips.cc/paper/2021/file/01b7575c38dac42f3cfb7d500438b875-Paper.pdf), DeepMind.

[Prompting Visual-Language Models for Efficient Video Understanding](https://arxiv.org/pdf/2112.04478.pdf), [[code](https://github.com/ju-chen/Efficient-Prompt)], SJTU.

[TiP-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling](https://arxiv.org/abs/2111.03930), [[code](https://github.com/gaopengcuhk/Tip-Adapter)], CUHK.