{"id":18887153,"url":"https://github.com/thunlp-mt/codis","last_synced_at":"2025-04-14T22:33:48.428Z","repository":{"id":223880089,"uuid":"760323208","full_name":"THUNLP-MT/CODIS","owner":"THUNLP-MT","description":"Repo for paper \"CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Models\".","archived":false,"fork":false,"pushed_at":"2024-10-14T18:19:58.000Z","size":89399,"stargazers_count":11,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-28T10:54:18.923Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/THUNLP-MT.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-20T07:51:18.000Z","updated_at":"2025-02-05T12:41:02.000Z","dependencies_parsed_at":"2024-06-04T11:20:44.216Z","dependency_job_id":null,"html_url":"https://github.com/THUNLP-MT/CODIS","commit_stats":null,"previous_names":["thunlp-mt/codis"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FCODIS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FCODIS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FCODIS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FCODIS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/THUNLP-MT","download_url":"https://codeload.github.com/THUNLP-MT/CODIS/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248973045,"owners_count":21191903,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T07:34:47.305Z","updated_at":"2025-04-14T22:33:43.404Z","avatar_url":"https://github.com/THUNLP-MT.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# \u003cimg src=\"./static/images/logo.png\" alt=\"logo\" width=\"3%\"\u003eCODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Models\r\n\r\n[**🌐 Homepage**](https://thunlp-mt.github.io/CODIS) | [**📖 arXiv**](https://arxiv.org/abs/2402.13607) | [**🤗 Dataset**](https://huggingface.co/datasets/CODIS/CODIS)\r\n\r\nThis repo contains the dataset and evaluation code for paper [CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Models](https://arxiv.org/abs/2402.13607).\r\n\r\n## Introduction\r\n\r\nIn certain situations, images need to be interpreted within a broader context. We introduce a new benchmark, named as **CODIS** (**CO**ntext-**D**ependent **I**mage di**S**ambiguation), designed to assess the ability of models to use context provided in free-form text to enhance visual comprehension.\r\n\r\n- Each image in CODIS contains inherent ambiguity that can only be resolved with additional context.\r\n- The questions are deliberately designed to highlight these ambiguities, requiring external context for accurate interpretation.\r\n- For every image-question pair, we provide two contexts in a free-form text format.\r\n\r\n![taxonomy](./static/images/taxonomy.jpg)\r\n\r\n## Evaluation\r\n\r\nPlease organize outputs of your models as follows.\r\n\r\n```\r\n[\r\n    {\r\n        \"id\": \"000\",\r\n        \"output\": {\r\n            \"output_1\": \"[YOUR OUTPUT TO CASE 000 GIVEN CONTEXT_1 HERE]\",\r\n            \"output_2\": \"[YOUR OUTPUT TO CASE 000 GIVEN CONTEXT_2 HERE]\"\r\n        }\r\n    },\r\n    {\r\n        \"id\": \"001\",\r\n        \"output\": {\r\n            \"output_1\": \"[YOUR OUTPUT TO CASE 001 GIVEN CONTEXT_1 HERE]\",\r\n            \"output_2\": \"[YOUR OUTPUT TO CASE 001 GIVEN CONTEXT_2 HERE]\"\r\n        }\r\n    },\r\n    ...\r\n]\r\n```\r\n\r\nA complete example of output file can be found at `data/output_example.json`.\r\n\r\nFor evaluation, please run the following command to calculate Acc_p and Acc_q.\r\n\r\n```bash\r\ncd data\r\n\r\n# your OpenAI API key\r\nexport OPENAI_API_KEY=[YOUR OPENAI API KEY HERE]\r\n# path to \"data.json\"\r\nexport ANSWER_PATH=./data.json\r\n# path to your output file\r\nexport OUTPUT_PATH=./output_example.json\r\n\r\npython eval.py\r\n```\r\n\r\n## Leaderboard\r\n\r\nWe report Acc_p scores based on human and GPT-4 evaluation. Models score only if their answers to a pair of queries are both correct. \r\n\r\n### Human Evaluation\r\n\r\n| Model          | Loc \u0026 Ori     | Temporal      | Cultural      | Attributes    | Relationships | Average       |\r\n|----------------|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|\r\nHuman            | 85.2          | 90.9          | 72.8          | 87.2          | 89.6          | 86.2          |\r\nGPT-4V           | 33.3          | 28.4          | 25.5          | 26.7          | 51.9          | 32.3          |\r\nGemini           | 21.4          | 29.5          | 21.3          | 24.0          | 34.6          | 26.1          |\r\nLLaVA-1.5-13B    | 6.0           | 4.2           | 10.6          | 14.7          | 13.5          | 9.1           |\r\nBLIP-2-11B       | 6.0           | 8.4           | 4.3           | 6.7           | 11.5          | 7.4           |\r\nInstructBLIP-13B | 6.0           | 2.1           | 4.3           | 4.0           | 7.7           | 4.5           |\r\nmPLUG-Owl-2-7B   | 13.1          | 9.5           | 6.4           | 12.0          | 19.2          | 11.9          |\r\nMiniGPT4-7B      | 10.7          | 3.2           | 0.0           | 12.0          | 13.5          | 7.9           |\r\nLLaVA-1.5-7B     | 11.9          | 5.3           | 4.3           | 9.3           | 7.7           | 7.9           |\r\nInstructBLIP-7B  | 1.2           | 7.4           | 0.0           | 4.0           | 11.5          | 4.8           |\r\nOtter-7B         | 2.4           | 5.3           | 4.3           | 0.0           | 5.8           | 3.4           |\r\nLLaVA-7B         | 2.4           | 6.3           | 0.0           | 1.3           | 5.8           | 3.4           |\r\nQwen-VL-Chat     | 3.6           | 3.2           | 0.0           | 1.3           | 9.6           | 3.4           |\r\nOpenFlamingo-7B  | 2.4           | 2.1           | 0.0           | 5.3           | 5.8           | 3.1           |\r\nBLIP-2-6.7B      | 0.0           | 1.1           | 2.1           | 2.7           | 7.7           | 2.3           |\r\n\r\n### GPT-4 Evaluation\r\n\r\n| Model          | Loc \u0026 Ori     | Temporal      | Cultural      | Attributes    | Relationships | Average       |\r\n|----------------|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|\r\nGPT-4V           | 33.3          | 28.4          | 21.3          | 25.3          | 50.0          | 31.2          |\r\nGemini           | 20.2          | 27.4          | 21.3          | 22.7          | 30.8          | 24.4          |\r\nLLaVA-1.5-13B    | 6.0           | 3.2           | 12.8          | 13.3          | 11.5          | 8.5           |\r\nBLIP-2-11B       | 6.0           | 10.5          | 4.3           | 6.7           | 11.5          | 8.0           |\r\nInstructBLIP-13B | 6.0           | 2.1           | 6.4           | 4.0           | 5.8           | 4.5           |\r\nmPLUG-Owl-2-7B   | 13.1          | 9.5           | 4.3           | 9.3           | 11.5          | 9.9           |\r\nMiniGPT4-7B      | 10.7          | 4.2           | 0.0           | 8.0           | 9.6           | 6.8           |\r\nLLaVA-1.5-7B     | 8.3           | 1.1           | 2.1           | 9.3           | 7.7           | 5.7           |\r\nInstructBLIP-7B  | 1.2           | 5.3           | 0.0           | 4.0           | 11.5          | 4.2           |\r\nOtter-7B         | 2.4           | 3.2           | 0.0           | 1.3           | 5.8           | 2.5           |\r\nLLaVA-7B         | 2.4           | 4.2           | 0.0           | 2.7           | 1.9           | 2.5           |\r\nQwen-VL-Chat     | 4.8           | 3.2           | 0.0           | 1.3           | 7.7           | 3.4           |\r\nOpenFlamingo-7B  | 2.4           | 2.1           | 0.0           | 5.3           | 5.8           | 3.1           |\r\nBLIP-2-6.7B      | 0.0           | 1.1           | 4.3           | 4.0           | 5.8           | 2.5           |\r\n\r\n## Citation\r\n\r\n```bibtex\r\n@article{luo2024codis,\r\n  title={CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Models},\r\n  author={Fuwen Luo and Chi Chen and Zihao Wan and Zhaolu Kang and Qidong Yan and Yingjie Li and Xiaolong Wang and Siyu Wang and Ziyue Wang and Xiaoyue Mi and Peng Li and Ning Ma and Maosong Sun and Yang Liu},\r\n  journal={arXiv preprint arXiv:2402.13607},\r\n  year={2024}\r\n}\r\n```\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthunlp-mt%2Fcodis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthunlp-mt%2Fcodis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthunlp-mt%2Fcodis/lists"}