{"id":21053883,"url":"https://github.com/mertyg/vision-language-models-are-bows","last_synced_at":"2025-09-25T18:51:41.665Z","repository":{"id":100582220,"uuid":"547451946","full_name":"mertyg/vision-language-models-are-bows","owner":"mertyg","description":"Experiments and data for the paper \"When and why vision-language models behave like bags-of-words, and what to do about it?\" Oral @ ICLR 2023","archived":false,"fork":false,"pushed_at":"2023-06-07T04:12:23.000Z","size":6307,"stargazers_count":277,"open_issues_count":7,"forks_count":18,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-05-20T08:07:59.666Z","etag":null,"topics":["blip","clip","compositionality","multimodal","pytorch","vision-language"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mertyg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-07T17:56:24.000Z","updated_at":"2025-05-16T12:54:34.000Z","dependencies_parsed_at":null,"dependency_job_id":"33521fa4-73b6-4a70-aa10-88b3b4c54513","html_url":"https://github.com/mertyg/vision-language-models-are-bows","commit_stats":{"total_commits":11,"total_committers":3,"mean_commits":"3.6666666666666665","dds":"0.36363636363636365","last_synced_commit":"81fdd20dc5b6f434aa5a4a971717bc80d1cedb0d"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mertyg/vision-language-models-are-bows","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mertyg%2Fvision-language-models-are-bows","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mertyg%2Fvision-language-models-are-bows/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mertyg%2Fvision-language-models-are-bows/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mertyg%2Fvision-language-models-are-bows/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mertyg","download_url":"https://codeload.github.com/mertyg/vision-language-models-are-bows/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mertyg%2Fvision-language-models-are-bows/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":276966325,"owners_count":25736757,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-25T02:00:09.612Z","response_time":80,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blip","clip","compositionality","multimodal","pytorch","vision-language"],"created_at":"2024-11-19T16:09:39.188Z","updated_at":"2025-09-25T18:51:41.626Z","avatar_url":"https://github.com/mertyg.png","language":"Python","funding_links":[],"categories":["Multimodal Perception \u0026 Understanding"],"sub_categories":["Language-Grounded Perception"],"readme":"# When and why vision-language models behave like bags-of-words, and what to do about it? (ICLR 2023 Oral)\n\n[![ICLR2023 Paper](https://img.shields.io/badge/paper-ICLR2023-brightgreen)](https://openreview.net/forum?id=KRLUvxh8uaX)  [![Medium Blog Post](https://raw.githubusercontent.com/aleen42/badges/master/src/medium.svg)](https://towardsdatascience.com/your-vision-language-model-might-be-a-bag-of-words-30b1beaef7f8) [![Colab](https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667)](https://colab.research.google.com/drive/1Rmn8CYXRFg4eC458vkBHwAdVKgS03e5D?usp=sharing)\n\n\nExperiments and data for the paper \"When and why vision-language models behave like bags-of-words, and what to do about it?\". \u003cbr\u003e\nThis paper got an Oral (notable-top-5%) at ICLR 2023! You can find our camera-ready version [here](https://openreview.net/forum?id=KRLUvxh8uaX).\n\n\n**Imporant Note**: Thank you for your interest. I apologize for the delay in releasing the code and the camera-ready version, I will do my best to make up for the missing bits as soon as possible. I am currently in Turkey after the devastating [Turkey-Syria earthquake](https://en.wikipedia.org/wiki/2023_Turkey%E2%80%93Syria_earthquake). Not only me, but also tens of thousands of people lost their families and homes. Please consider [donating](https://ahbap.org/), and at the very least please ask your friends with connections to the regions how they are doing. \n\nBelow we give details about how to easily use our dataset and models, and reproduce our experiments.\n\n# ARO Benchmark\n## Visual Genome Relation \u0026 Attribution Datasets\nIt's very easy to use VG-Relation and VG-Attribution datasets. Here's an example:\n```python\nimport clip\nfrom dataset_zoo import VG_Relation, VG_Attribution\n\nmodel, image_preprocess = clip.load(\"ViT-B/32\", device=\"cuda\")\n\nroot_dir=\"/path/to/aro/datasets\"\n# Setting download=True will download the dataset to `root_dir` if it's not already there. \n# For VG-R and VG-A, this is a 1GB zip file that is a subset of GQA.\n\nvgr_dataset = VG_Relation(image_preprocess=preprocess, download=True, root_dir=root_dir)\nvga_dataset = VG_Attribution(image_preprocess=preprocess, download=True, root_dir=root_dir)\n\n# Do anything with the dataset. Each item will look like this : \n# item = {\"image_options\": [image], \"caption_options\": [false_caption, true_caption]}\n```\n\n## COCO-Order and Flickr30k-Order Datasets\nThese datasets require the COCO and Flickr30k retrieval datasets. We provided the interface to download COCO (e.g. set `download=True` in the constructor), however, for Flickr30k, you need to sign up and download it yourself. You can find the Flickr30k retrieval dataset [here](https://forms.illinois.edu/sec/229675).\n\n```python\nfrom dataset_zoo import COCO_Order, Flickr30k_Order\n\ncoco_order_dataset = COCO_Order(image_preprocess=preprocess, download=True, root_dir=root_dir) \nflickr_order_dataset = Flickr30k_Order(image_preprocess=preprocess, root_dir=root_dir)\n```\n\n\n# Quick reproducibility\nSee the notebook in `notebooks/` for a quick way to reproduce some of the results in the paper. We provide a notebook to reproduce the VG-Relation and VG-Attribution datasets [here](notebooks/Replicate%20ARO!%20VG-Relation%2C%20VG-Attribution.ipynb).\n\n## Models\nWe experiment with a bunch of models here, and let us know if you have any other you would like to add here. You can find BLIP, CLIP, Flava, and XVLM. Please see `model_zoo/` folder for more details. This work is heavily inspired from, and would not be possible without the awesome repos for [BLIP](https://github.com/salesforce/BLIP), [CLIP](https://github.com/openai/CLIP), [Flava](https://huggingface.co/docs/transformers/model_doc/flava), [OpenCLIP](https://github.com/mlfoundations/open_clip), and [XVLM](https://github.com/zengyan-97/X-VLM). A huge, huge thanks to them for open-sourcing their models / implementations! Here's a summary of what we have now: \n\nModel Name | Model File in this Repo | Repo |\n--- | --- | --- |\nBLIP | [BLIP implementation](model_zoo/blip_models.py) | https://github.com/salesforce/BLIP |\nCLIP | [CLIP implementation](model_zoo/clip_models.py) | https://github.com/openai/CLIP |\nFlava | [Flava implementation](model_zoo/flava.py) | https://huggingface.co/facebook/flava-full |\nXVLM | [XVLM implementation](model_zoo/xvlm_models.py) | https://github.com/zengyan-97/X-VLM |\nNegCLIP | NegCLIP was trained with a fork of the `open_clip` repo. Find the ckpt info [here](model_zoo/__init__.py#L66)| https://github.com/vinid/open_clip |\nCOCA \u0026 CLIP on LAION | We added the usage of the other models in the open_clip repo.| https://github.com/mlfoundations/open_clip |\n\n\n## ARO Results\n\n\n## Order-Perturbed Retrieval Results\n\n\n## NegCLIP Training\nWe trained the NegCLIP with a fork of the `open_clip` repo. You can find the fork [here](https://github.com/vinid/open_clip). Our modifications are super minor and you will find an detailed description of the main edits [here](https://github.com/vinid/neg_clip#negclip-implementation).\n\nWe plan to add support for the distributed setting in the future. However, we trained the model using a single GPU (which is quite a bit of a limitation). Here's the command to reproduce results:\n```base\nCUDA_VISIBLE_DEVICES=0 python -m training.main \\\n    --train-data=\"./mscoco_with_negatives_training.csv\" \\\n    --batch-size=256 \\\n    --epochs=5 \\\n    --name=\"negclip_256_1e-6\" \\\n    --lr=1e-6 \\\n    --val-data=\"./mscoco_with_negatives_valid.csv\"  \\\n    --logs=\"./logs/negCLIP/\" \\\n    --pretrained=\"openai\" \\\n    --model=\"ViT-B-32\"\\\n    --workers 14 \\\n    --warmup 50\n```\nNote here that `batch_size=256` would result in a matrix of size `512x1024` with negatives.\n\n\n# Citation\nIf you use this code or data, please consider citing our paper:\n\n```\n@inproceedings{\n  yuksekgonul2023when,\n  title={When and why Vision-Language Models behave like  Bags-of-Words, and what to do about it?},\n  author={Mert Yuksekgonul and Federico Bianchi and Pratyusha   Kalluri and Dan Jurafsky and James Zou},\n  booktitle={International Conference on Learning Representations},\n  year={2023},\n  url={https://openreview.net/forum?id=KRLUvxh8uaX}\n}\n```\n\n\n## TODO\n\u003cdetails\u003e\n\u003csummary\u003e Current TODO List.\u003c/summary\u003e\n\n| Name | Description | Status |\n| --- | --- | --- |\n| Add support for distributed training | We trained NegCLIP with a single GPU, and we plan to add support for distributed training in the future. | :white_check_mark: |\n| Add negative generation | How to generate negatives for negclip. This could also be on the forked repo. | :white_check_mark: |\n\n\u003c/details\u003e\n\n# Contact \nPlease let us know if you have further questions or comments. You can reach out to me at `merty@stanford.edu`. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmertyg%2Fvision-language-models-are-bows","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmertyg%2Fvision-language-models-are-bows","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmertyg%2Fvision-language-models-are-bows/lists"}