awesome-multimodal-data-recipe
Curated collection of multimodal data synthesis methods, covering papers, datasets, and best practices for vision-language model training
https://github.com/OpenRaiser/awesome-multimodal-data-recipe
Last synced: 11 days ago
JSON representation
-
๐ Acknowledgments
-
Contribution Guidelines
-
-
๐ Benchmark Datasets
-
๐ง Cross-Domain Methodology Insights
-
๐ Foundation Model Mid-training & Data Curation
-
-
๐ข Industrial & Open-Source Data Synthesis
-
Alibaba - MMEvol (2024)
-
Alibaba - Qwen-VL Series
-
Baidu - Qianfan-VL
-
ByteDance & NTU - LLaVA-OneVision
-
ByteDance - Oasis
-
LLaVA Series (Wisconsin-Madison & Microsoft)
-
Microsoft - Florence-2
-
Tencent Hunyuan - Bee, HoneyPipe & DataStudio
-
-
๐ Methods by Image Processing Type
-
๐งฉ Compositionality / Preference-Guided Synthesis
- (arXiv 2503.01167) - **CVPR 2025**
- Dataset & Code
- tripletclip.github.io
- Project - Code, MyCanvas Dataset (2,684 images), Benchmark Metrics
- tripletclip.github.io
-
โ๏ธ Image Editing (Method + Data)
-
๐จ Image Generation - Synthesizing New Visual Content
- (OpenReview ICLR 2025)
- Code & Models
- (arxiv 2407.08739)
- Code & Data
- (arxiv 2506.18095)
- Code & Data
- (arXiv 2505.20275) - **ICCV 2025**
- GitHub
- https://yyyyyxie.github.io/textflux-site/
- Code, Data & Models
- Code - modal-self-instruct.github.io)
- yueyang1996.github.io/cosyn
- Code
- github.com/modelscope/data-juicer/tree/ImgDiff
- yueyang1996.github.io/cosyn
-
Image-Invariant Text Enhancement
- (arxiv 2311.12793)
- Dataset - XComposer/tree/main/projects/ShareGPT4V)
- (arxiv 2307.04087)
- Dataset
- (arxiv 2310.20550)
- Code
- (arxiv 2406.07502)
- Code - textualization/)
- (arxiv 2306.17107)
- Code
- (arxiv 2402.11684)
- (arxiv 2510.15040)
- (arxiv 2308.01907)
- Dataset - seeing)
- ACL Anthology
- arXiv:2311.10774
- arXiv:2411.19930
- HuggingFace
- arXiv:2403.00231
- Code - 10M)
- Code
- Code and Data
- Code, Data, and Models
- Code, Data, and Model
- Dataset & Generation Code
- github.com/Yu-xm/Unicorn
- GitHub - Code, MANTIS-INSTRUCT Dataset (721K), Models
- GitHub - Code, MMDU Benchmark (110 dialogues), MMDU-45K Dataset (45K samples)
- Code and Data
- Dataset
- Project Page
- GitHub
- arXiv:2402.12226
- GitHub
- Code
- github.com/google/imageinwords
- Code
- Code
- arXiv:2502.09925
- GitHub
- Code - textualization/)
-
๐งช Interleaved Image-Text ยท Coherence & Consistency
- (arXiv 2406.10462) - **CVPR 2025**
- GitHub - Code, SMIR Dataset (160K samples), SMIR-BENCH (200 examples)
- arXiv:2501.00958
-
๐ญ Think with Image
-
-
๐ฆ Notable Multimodal Datasets
-
๐ Domain-Specific & Knowledge-Oriented Datasets
-
๐ฆ Interleaved Image-Text Datasets
-
๐ Large-Scale General Training Datasets
-
-
๐ Resources
-
โญ Star History
-
Contribution Guidelines
- ![Star History Chart - history.com/#opendatalab-raiser/awesome-multimodal-data-recipe&Date)
- ![Star History Chart - history.com/#opendatalab-raiser/awesome-multimodal-data-recipe&Date)
-
-
๐ฏ VLM Self-Improvement & Reinforcement Learning
-
Image-Invariant Text Enhancement
-
Programming Languages
Categories
๐ Methods by Image Processing Type
70
๐ข Industrial & Open-Source Data Synthesis
11
๐ฆ Notable Multimodal Datasets
10
๐ Benchmark Datasets
6
๐ Resources
6
๐ง Cross-Domain Methodology Insights
3
๐ Acknowledgments
2
๐ฏ VLM Self-Improvement & Reinforcement Learning
2
โญ Star History
2
Sub Categories
Image-Invariant Text Enhancement
43
๐จ Image Generation - Synthesizing New Visual Content
15
๐งฉ Compositionality / Preference-Guided Synthesis
5
๐ Large-Scale General Training Datasets
4
โ๏ธ Image Editing (Method + Data)
4
๐ฆ Interleaved Image-Text Datasets
4
Contribution Guidelines
4
LLaVA Series (Wisconsin-Madison & Microsoft)
3
๐งช Interleaved Image-Text ยท Coherence & Consistency
3
๐ Foundation Model Mid-training & Data Curation
3
๐ฏ Instruction-Following Benchmarks
3
๐ Task-Specific Benchmarks
3
๐ Important Blogs
2
๐ Survey Papers (Covering Data Synthesis)
2
Microsoft - Florence-2
2
๐ Domain-Specific & Knowledge-Oriented Datasets
2
๐ญ Think with Image
2
ByteDance & NTU - LLaVA-OneVision
1
ByteDance - Oasis
1
Baidu - Qianfan-VL
1
Alibaba - Qwen-VL Series
1
Alibaba - MMEvol (2024)
1
Tencent Hunyuan - Bee, HoneyPipe & DataStudio
1
๐ก Influential Inspirational Work
1
๐ Related Awesome Lists
1
Keywords
t2i
1
image-to-text
1
image-text
1
image-descriptions
1
image-captioning
1
i2t
1
human-annotation
1
evaluation
1
detailed-descriptions
1
detailed-annotations
1
dataset-generation
1
dataset
1
speech-processing
1
robotics
1
representation-learning
1
reinforcement-learning
1
reading-list
1
natural-language-processing
1
multimodal-learning
1
machine-learning
1
healthcare
1
deep-learning
1
computer-vision
1