{"id":15027837,"url":"https://github.com/nvlabs/prismer","last_synced_at":"2025-05-16T07:00:26.160Z","repository":{"id":126002088,"uuid":"608406286","full_name":"NVlabs/prismer","owner":"NVlabs","description":"The implementation of \"Prismer: A Vision-Language Model with Multi-Task Experts\".","archived":false,"fork":false,"pushed_at":"2024-01-17T21:55:00.000Z","size":4453,"stargazers_count":1310,"open_issues_count":0,"forks_count":73,"subscribers_count":16,"default_branch":"main","last_synced_at":"2025-04-08T16:06:34.108Z","etag":null,"topics":["image-captioning","language-model","multi-modal-learning","multi-task-learning","vision-and-language","vision-language-model","vqa"],"latest_commit_sha":null,"homepage":"https://shikun.io/projects/prismer","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NVlabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-02T00:20:34.000Z","updated_at":"2025-03-16T09:20:22.000Z","dependencies_parsed_at":"2024-01-17T22:49:38.247Z","dependency_job_id":"6935e383-a7b4-4446-a4ec-2f88b1443733","html_url":"https://github.com/NVlabs/prismer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVlabs%2Fprismer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVlabs%2Fprismer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVlabs%2Fprismer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVlabs%2Fprismer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NVlabs","download_url":"https://codeload.github.com/NVlabs/prismer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254485025,"owners_count":22078764,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["image-captioning","language-model","multi-modal-learning","multi-task-learning","vision-and-language","vision-language-model","vqa"],"created_at":"2024-09-24T20:07:08.871Z","updated_at":"2025-05-16T07:00:26.131Z","avatar_url":"https://github.com/NVlabs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Prismer\n[![arXiv](https://img.shields.io/badge/arXiv-2303.02506-b31b1b.svg)](https://arxiv.org/abs/2303.02506)\n [![Hugginface Space](https://huggingface.co/datasets/huggingface/badges/raw/main/open-in-hf-spaces-sm-dark.svg)](https://huggingface.co/spaces/shikunl/prismer)\n\n\nThis repository contains the source code of **Prismer** and **PrismerZ** from the paper, [Prismer: A Vision-Language Model with Multi-Task Experts](https://shikun.io/projects/prismer). Check out our official demo at [HuggingFace Space](https://huggingface.co/spaces/lorenmt/prismer) and a third-party demo at [Replicate](https://replicate.com/cjwbw/prismer).\n\n\u003cimg src=\"helpers/intro.png\"  width=\"100%\"/\u003e\n\n### Update Log\n- **Apr. 3, 2023**: Improve HF Space demo with half precision inference and md5sum image check.\n- **Mar. 22, 2023**: Fix tensor mismatch problem for the updated `transformers` package.\n- **Mar. 13, 2023**: Add official HF Space demo. \n\n## Get Started\nThe implementation is based on `PyTorch 1.13`, and highly integrated with Huggingface [`accelerate`](https://github.com/huggingface/accelerate) toolkit for readable and optimised multi-node multi-gpu training.\n\nFirst, let's install all package dependencies by running\n```bash\npip install -r requirements.txt\n```\n\n### Prepare Accelerator Config\nThen we generate the corresponding `accelerate`  config based on your training server configuration. For both single-node multi-gpu and multi-node multi-gpu training, simply run and follow the instructions with,\n```bash\naccelerate config\n```\n\n## Datasets\n\n### Pre-training\nWe pre-train Prismer/PrismerZ with a combination of five widely used image-alt/text datasets, with pre-organised data lists provided below.\n- [COCO 2014](https://www.dropbox.com/s/6btr8hz5n1e1q4d/coco_karpathy_train.json?dl=0): the Karpathy training split (which will also be used for fine-tuning).\n- [Visual Genome](https://www.dropbox.com/s/kailbaay0sqraxc/vg_caption.json?dl=0): the official Visual Genome captioning dataset.\n- [CC3M + SGU](https://www.dropbox.com/s/xp2nuhc88f1czxm/filtered_cc3m_sbu.json?dl=0): filtered and re-captioned by BLIP-Large.\n- [CC12M](https://www.dropbox.com/s/th358bb6wqkpwbz/filtered_cc12m.json?dl=0): filtered and re-captioned by BLIP-Large.\n\nThe web datasets (CC3M, SGU, CC12M) is composed with image urls. It is highly recommended to use [img2dataset](https://github.com/rom1504/img2dataset), a highly optimised toolkit for large-scale web scraping to download these images. An example bash script of using `img2dataset` to download `cc12m` dataset is provided below.\n```bash\nimg2dataset --url_list filtered_cc12m.json --input_format \"json\" --url_col \"url\" --caption_col \"caption\" --output_folder cc12m --processes_count 16 --thread_count 64 --image_size 256\n```\n\n*Note: It is expected that the number of downloaded images is less than the number of images in the json file, because some urls might not be valid or require long loading time.*\n\n### Image Captioning / VQA\nWe evaluate image captioning performance on two datasets, COCO 2014 and NoCaps; and VQA performance on VQAv2 dataset. In VQA tasks, we additionally augment the training data with Visual Genome QA, following BLIP. Again, we have prepared and organised the training and evaluation data lists provided below.\n\n- [Image Captioning](https://www.dropbox.com/sh/quu6v5hzdetjcdz/AACze0_h6BO8LJmSsEq4MM8-a?dl=0): including COCO (Karpathy Split) and NoCaps.\n- [VQAv2](https://www.dropbox.com/sh/hqtxl1k8gkbhhoi/AACiax5qi7no3pJgO1E57Xefa?dl=0): including VQAv2 and VG QA.\n\n## Generating Expert Labels\nBefore starting any experiments with Prismer, we need to first pre-generate the modality expert labels, so we may construct a multi-label dataset. In `experts` folder, we have included all 6 experts we introduced in our paper. We have organised each expert's codebase with a shared and simple API.\n\n*Note: Specifically for segmentation experts, please first install deformable convolution operations by `cd experts/segmentation/mask2former/modeling/pixel_decoder/ops` and run `sh make.sh`.*\n\nTo download pre-trained modality experts, run\n```bash\npython download_checkpoints.py --download_experts=True\n```\n\nTo generate the expert labels, simply edit the `configs/experts.yaml` with the corresponding data paths, and run\n```bash\nexport PYTHONPATH=.\naccelerate launch experts/generate_{EXPERT_NAME}.py\n```\n*Note: Expert label generation is only required for Prismer models, not for PrismerZ models.*\n\n## Experiments\nWe have provided both Prismer and PrismerZ for pre-trained checkpoints (for zero-shot image captioning), as well as fined-tuned checkpoints on VQAv2 and COCO datasets. With these checkpoints, it should be expected to reproduce the exact performance listed below.\n\n| Model          | Pre-trained [Zero-shot] | COCO [Fine-tuned]   | VQAv2 [Fine-tuned] |\n|----------------|-------------------------|---------------------|-------------------|\n| PrismerZ-BASE  | COCO CIDEr [109.6]      | COCO CIDEr [133.7]\t | test-dev [76.58]  |\n| Prismer-BASE   | COCO CIDEr [122.6]      | COCO CIDEr [135.1]\t | test-dev [76.84]  |\n| PrismerZ-LARGE | COCO CIDEr [124.8]      | COCO CIDEr [135.7]\t | test-dev [77.49]  |\n| Prismer-LARGE  | COCO CIDEr [129.7]      | COCO CIDEr [136.5]\t | test-dev [78.42]  |\n\nTo download pre-trained/fined-tuned checkpoints, run\n```bash\n# to download all model checkpoints (12 models in total)\npython download_checkpoints.py --download_models=True\n\n# to download specific checkpoints (Prismer-Base for fine-tuned VQA) in this example\npython download_checkpoints.py --download_models=\"vqa_prismer_base\"\n```\n\n\n*Note: Remember to install java via `sudo apt-get install default-jre` which is required to run the official COCO caption evaluation scripts.*\n\n\n### Evaluation\nTo evaluate the model checkpoints, please run\n```bash\n# zero-shot image captioning (remember to remove caption prefix in the config files)\naccelerate launch train_caption.py --exp_name {MODEL_NAME} --evaluate\n\n# fine-tuned image captioning\naccelerate launch train_caption.py --exp_name {MODEL_NAME} --from_checkpoint --evaluate\n\n# fine-tuned VQA\naccelerate launch train_vqa.py --exp_name {MODEL_NAME} --from_checkpoint --evaluate\n```\n\n### Training / Fine-tuning\nTo pre-train or fine-tune any model with or without checkpoints, please run\n```bash\n# to train/fine-tuning from scratch\naccelerate launch train_{TASK}.py --exp_name {MODEL_NAME}\n\n# to train/fine-tuning from the latest checkpoints (saved every epoch)\naccelerate launch train_{TASK}.py --exp_name {MODEL_NAME} --from_checkpoint \n```\n\nWe have also included model sharding in the current training script via PyTorch's official [FSDP plugin](https://pytorch.org/tutorials/intermediate/FSDP_tutorial.html). With the same training commands, additionally add `--shard_grad_op` for ZeRO-2 Sharding (Gradients + Optimiser States), or `--full_shard` for ZeRO-3 Sharding (ZeRO-2 + Network Parameters). \n\n*Note: You should expect the error range for VQAv2 Acc. to be less than 0.1; for COCO/NoCAPs CIDEr score to be less than 1.0.*\n\n## A Minimal Example\nFinally, we have offered a minimal example to perform image captioning in a single GPU with our fine-tuned Prismer/PrismerZ checkpoint. Simply put your images under `helpers/images` (support `.jpg`, `.jpeg`, and `.png` images), and run\n```bash\npython demo.py --exp_name {MODEL_NAME}\n```\n\nYou then can see all generated modality expert labels in the `helpers/labels` folder and the generated captions in the `helpers/images` folder.\n\nParticularly for the Prismer models, we have also offered a simple script to prettify the generated expert labels. To prettify and visualise the expert labels as well as its predicted captions, run\n```bash\npython demo_vis.py\n```\n\n*Note: Remember to set up the corresponding config in the `configs/caption.yaml` demo section. The default demo model config is for Prismer-Base.*\n\n## Citation\n\nIf you found this code/work to be useful in your own research, please considering citing the following:\n\n\n```bibtex\n@article{liu2024prismer,\n    title={Prismer: A Vision-Language Model with Multi-Task Experts},\n    author={Liu, Shikun and Fan, Linxi and Johns, Edward and Yu, Zhiding and Xiao, Chaowei and Anandkumar, Anima},\n    journal={Transactions on Machine Learning Research},\n    year={2024}\n}\n```\n\n## License\nCopyright © 2023, NVIDIA Corporation. All rights reserved.\n\nThis work is made available under the Nvidia Source Code License-NC. \n\nThe model checkpoints are shared under CC-BY-NC-SA-4.0. If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.\n\nFor business inquiries, please visit our website and submit the form: [NVIDIA Research Licensing](https://www.nvidia.com/en-us/research/inquiries/).\n\n## Acknowledgement\nWe would like to thank all the researchers who open source their works to make this project possible. [@bjoernpl](https://github.com/bjoernpl) for contributing an automated checkpoint download script.\n\n## Contact\nIf you have any questions, please contact `sk.lorenmt@gmail.com`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnvlabs%2Fprismer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnvlabs%2Fprismer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnvlabs%2Fprismer/lists"}