{"id":28447826,"url":"https://github.com/opengvlab/mm-interleaved","last_synced_at":"2025-06-30T13:32:30.695Z","repository":{"id":217938901,"uuid":"744938845","full_name":"OpenGVLab/MM-Interleaved","owner":"OpenGVLab","description":"MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer ","archived":false,"fork":false,"pushed_at":"2024-04-03T03:09:27.000Z","size":3779,"stargazers_count":227,"open_issues_count":7,"forks_count":12,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-06-06T12:07:19.271Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OpenGVLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-01-18T10:08:08.000Z","updated_at":"2025-06-06T09:51:18.000Z","dependencies_parsed_at":"2024-02-01T08:26:47.944Z","dependency_job_id":"39d7f747-aa85-4cd1-9c9c-45ad05ea7c95","html_url":"https://github.com/OpenGVLab/MM-Interleaved","commit_stats":null,"previous_names":["opengvlab/mm-interleaved"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/OpenGVLab/MM-Interleaved","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenGVLab%2FMM-Interleaved","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenGVLab%2FMM-Interleaved/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenGVLab%2FMM-Interleaved/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenGVLab%2FMM-Interleaved/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OpenGVLab","download_url":"https://codeload.github.com/OpenGVLab/MM-Interleaved/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenGVLab%2FMM-Interleaved/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262783328,"owners_count":23363516,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-06T12:07:19.475Z","updated_at":"2025-06-30T13:32:30.677Z","avatar_url":"https://github.com/OpenGVLab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer\n\nThis is the official repository of **MM-Interleaved**: an end-to-end generative model for interleaved image-text data.\n\n![](docs/figures/diagram.png)\n\n## Introduction\n\n**MM-Interleaved** is a new end-to-end generative model for interleaved image-text modeling. It introduces a novel fine-grained multi-modal feature synchronizer named **MMFS**, allowing it to recognize multi-scale high-resolution features across multiple images for producing accurate textual descriptions and visually consistent images auto-regressively. Please refer to our [paper](https://arxiv.org/pdf/2401.10208.pdf) for more technical details.\n\n![](docs/figures/arch.png)\n\nThe model is pretrained on a mixture of publicly available datasets, achieving superior zero-shot performance on various evaluation benchmarks of multi-modal comprehension and generation. It can be further fine-tuned for different downstream tasks, such as visual question answering, image captioning, referring expression grounding, text-to-image generation, segmentation-to-image translation, and visual storytelling. \n\n## Examples\n\n\u003cdiv align=center\u003e\u003cimg src=\"docs/figures/text_generation.png\" style=\"zoom:60%;\" /\u003e\u003c/div\u003e\n\n\u003cdiv align=center\u003e\u003cimg src=\"docs/figures/image_generation.png\" style=\"zoom:75%;\" /\u003e\u003c/div\u003e\n\n\u003cdiv align=center\u003e\u003cimg src=\"docs/figures/segment.png\" width=70% height=70%/\u003e\u003c/div\u003e\n\n\u003cdiv align=center\u003e\u003cimg src=\"docs/figures/interleaved.png\" width=50% height=50% /\u003e\u003c/div\u003e\n\n## Setup\n\nStep 1, clone the repository and use `pip` to install all required packages.\n\n```shell\ngit clone https://github.com/OpenGVLab/MM-Interleaved\ncd MM-Interleaved\npip install -r requirements.txt\n# install `MultiScaleDeformableAttention` module\ncd mm_interleaved/models/utils/ops\npython setup.py install\n```\n\nStep 2, download all pretrained model components from huggingface into the `assets/` directory by running the following commands:\n\n```\npython mm_interleaved/scripts/download_hf_models.py\n```\n\nThe pretrained model weights of MM-Interleaved (without instruction tuning or SFT) are provided at this [link](https://huggingface.co/OpenGVLab/MM-Interleaved/tree/main/mm_interleaved_pretrain), under Llama's [model license](https://github.com/facebookresearch/llama/blob/main/LICENSE). You can download them to `OUTPUT/mm_interleaved_pretrain` and use it for further inference, zero-shot evaluation or finetuning tasks.\n\n\n\n## Inference\n\nFor inference, we provide an example inference script `./inference.py` and the corresponding configuration file `./mm_interleaved/configs/release/mm_inference.yaml`, which natively support interleaved image and text generation. Simply run the following command:\n\n```\npython -u inference.py --config_file=mm_interleaved/configs/release/mm_inference.yaml\n```\n\nThe generation results will be automatically saved under the `./OUTPUT/mm_inference ` directory.\n\nThe interleaved inputs data is from `./docs/examples/annt.json`  by default. Feel free to modify it for custom generation. Below is an example of the interleaved input context:\n\n```\n{\n    // list of input text sentences\n    \"sentences\": [\n        \"a kitchen is shown with a variety of items on the counters.\"\n    ], \n    // list of input image paths\n    \"images\": [\n        \"./assets/dataset/coco/val2014/COCO_val2014_000000384213.jpg\"\n    ], \n    // list of corresponding sentence indexs for \"images\"\n    \"sentence_ixs\": [\n        0 \n    ], \n    // list of the relative order of images and their corresponding sentences \n    \"image_first\": [\n        false \n    ], \n    // the initial generation mode for the first iteration, which can be either \"generate_images\" or \"generate_texts\"\n    \"generate_mode\": \"generate_images\",\n    // the number of total iteration\n    \"num_iter\": 1\n}\n```\n\nNote that if the `\"generate_mode\"` is set to `\"generate_images\"`, then a padding image should be appended after the last sentence serving as a placeholder. Currently the inputs should contain at least 1 image (including the padding one if exists).\n\n\n\n## Evaluation\n\nWe provide the script `./evaluate.py` and the corresponding configuration file `./mm_interleaved/configs/release/mm_eval.yaml` for zero-shot evaluation of our pretrained model on all benchmarks mentioned in our paper. We also upload all the annotation files [here](https://huggingface.co/OpenGVLab/MM-Interleaved/tree/main/assets/datasets) for easy access.\n\nThe evaluation is conducted by default under the slurm distributed environment. Please first download the pretrained model weights and the evaluation data per dataset under the same path specified in the configuration file. Then run the following commands:\n\n``` \nbash slurm_run.sh ${GPUS} ${GPUS_PER_NODE} ${JOB_NAME} ${QUOTATYPE} ${PARATITION} evaluate.py ./mm_interleaved/configs/release/mm_eval.yaml\n```\n\nThe file structure for all evaluation datasets is expected to be something like this:\n\n```\n├── coco\n│   ├── annotations\n│   │   ├── captions_val2014.json\n│   │   ├── captions_val2017.json\n│   │   ├── coco_karpathy_test_gt.json\n│   │   ├── coco_karpathy_test.json\n│   │   ├── coco_karpathy_val_gt.json\n│   │   └── coco_karpathy_val.json\n│   ├── test2014\n│   ├── val2014\n│   └── val2017\n├── flickr30k\n│   ├── flickr30k-images\n│   └── test1k.token.coco_format\n├── image2paragraph\n│   ├── annotations\n│   │   ├── paragraphs_coco.json\n│   │   ├── paragraphs_v1.json\n│   │   ├── test_split.json\n│   │   └── val_split.json\n│   └── images\n├── lncoco\n│   ├── coco_val_captions.jsonl\n├── nocaps\n│   ├── images\n│   └── nocaps_val_4500_captions.json\n├── OK-VQA\n│   ├── mscoco_val2014_annotations.json\n│   └── OpenEnded_mscoco_val2014_questions.json\n├── textvqa\n│   ├── textvqa_val_annotations.json\n│   ├── textvqa_val_questions.json\n│   └── train_images\n├── visdial\n│   ├── visdial_1.0_val_dense_annotations.json\n│   ├── visdial_1.0_val.json\n│   └── VisualDialog_val2018\n├── VizWiz\n│   ├── val\n│   └── val.json\n└── VQAv2\n    ├── v2_mscoco_val2014_annotations.json\n    └── v2_OpenEnded_mscoco_val2014_questions.json\n```\n\n\n\n## Pre-training\n\nWe also provide the pre-training script `./train.py`  and the corresponding configuration file `mm_interleaved/configs/release/mm_pretrain.yaml`. By default, we use DeepSpeed ZeRO-1 for efficient distributed training under slurm environment. The training engine is implemented based on the huggingface's trainer.\n\nWe use webdataset for efficient data loading. For MMC4 dataset, simply download all the images and annotation files under the same directory specified in the configuration file. The raw annotation file with the suffix `.jsonl.zip` can be directly processed on the fly without further modification.  For image-text pair datasets such as LAION-COCO and LAION-En, each annotation file is expected to have ~10,000 lines of image-text pair data. Each line is a json dictionary containing two key-value pairs, i.e. the \"image\" and corresponding \"caption\". An example annotation file is provided at `./docs/examples/0000001.txt`. After preparing all the pretrained data, please modify the data paths as well as other relative arguments in the configuration file.\n\nMoreover, we support zero-shot evaluation for multiple datasets during training as well. To achieve this, modify the `data.val` part of the training configuration file and include the configs of the evaluation datasets which you want to monitor in real time during training.\n\nThe pre-training command is:\n\n```\nbash slurm_run.sh ${GPUS} ${GPUS_PER_NODE} ${JOB_NAME} ${QUOTATYPE} ${PARATITION} train.py ./mm_interleaved/configs/release/mm_pretrain.yaml\n```\n\n\n\n## Schedule\n\n- [x] Release inference \u0026 evaluation code\n\n- [x] Release pre-training code\n\n- [x] Release pre-training model weights\n\n- [ ] Release finetuning code\n\n\n\n## Acknowledgements\n\nWe thank the great work from [OpenFlamingo](https://github.com/mlfoundations/open_flamingo), [transformers](https://github.com/huggingface/transformers), [diffusers](https://github.com/huggingface/diffusers), [LLaMA](https://github.com/facebookresearch/llama), [CLIP](https://github.com/openai/CLIP), [BLIP](https://github.com/salesforce/BLIP), and [ViT-Adapter](https://github.com/czczup/ViT-Adapter).\n\n\n\n## Citing\n\nIf you find MM-Interleaved helpful, please consider giving this repository a star :star: and citing in your publications.\n\n```BibTeX\n@article{tian2024mminterleaved,\n  title={MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer}, \n  author={Tian, Changyao and Zhu, Xizhou and Xiong, Yuwen and Wang, Weiyun and Chen, Zhe and Wang, Wenhai and Chen, Yuntao and Lu, Lewei and Lu, Tong and Zhou, Jie and Li, Hongsheng and Qiao, Yu and Dai, Jifeng},\n  journal={arXiv preprint arXiv:2401.10208},\n  year={2024},\n}\n```\n\n\n\n## License\n\nThis project is released under the [Apache 2.0 license](https://github.com/OpenGVLab/MM-Interleaved/blob/main/LICENSE). Parts of this project contain code and models from other sources, which are subject to their respective licenses.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopengvlab%2Fmm-interleaved","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopengvlab%2Fmm-interleaved","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopengvlab%2Fmm-interleaved/lists"}