{"id":18887105,"url":"https://github.com/thunlp-mt/actiview","last_synced_at":"2025-09-04T08:32:55.450Z","repository":{"id":259885307,"uuid":"868485233","full_name":"THUNLP-MT/ActiView","owner":"THUNLP-MT","description":null,"archived":false,"fork":false,"pushed_at":"2024-12-20T05:36:53.000Z","size":694155,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-12-20T06:26:29.914Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/THUNLP-MT.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-06T14:02:26.000Z","updated_at":"2024-12-20T05:36:57.000Z","dependencies_parsed_at":"2024-12-20T06:24:00.322Z","dependency_job_id":"973ba1cc-e254-4324-8e1c-2439714afca1","html_url":"https://github.com/THUNLP-MT/ActiView","commit_stats":null,"previous_names":["thunlp-mt/actiview"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FActiView","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FActiView/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FActiView/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FActiView/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/THUNLP-MT","download_url":"https://codeload.github.com/THUNLP-MT/ActiView/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":231945439,"owners_count":18449886,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T07:34:10.005Z","updated_at":"2024-12-31T05:18:55.195Z","avatar_url":"https://github.com/THUNLP-MT.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ActiView: Evaluating Active Perception Ability for Multimodal Large Language Models\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"./figures/figure1.png\" alt=\"Image\" width=60% \u003e\u003c/p\u003e\n\n\u003cdiv align=\"center\"\u003e\n    \u003ca href=\"https://wangphoebe.github.io/actiview_homepage/\"\u003e🌐\u003cstrong\u003eHomepage\u003c/strong\u003e\u003c/a\u003e | \u003ca href=\"https://arxiv.org/abs/2410.04659\"\u003e📖\u003cstrong\u003earXiv\u003c/strong\u003e\u003c/a\u003e | \u003ca href=\"https://huggingface.co/datasets/wangphoebe/ActiView\"\u003e🤗\u003cstrong\u003eDataset\u003c/strong\u003e\u003c/a\u003e\n\u003c/div\u003e\n\n  \n## Timeline\n\n📢 [2024-10-23] Benchmark and evaluation toolkit (will be updated soon, does not support single-image models) released.\n\n📢 [2024-10-14] Homepage released.\n\n📢 [2024-10-07] Paper and repo released.  \n\n## Project Overview\nThis repository contains all the necessary materials, including:\n- **Datasets** for active perception evaluation\n- **Toolkits** for model evaluation\n- **Results** and analysis from the paper\n\n## Leaderboard\n\n### Results of Zooming and Shifting Pipelines\n\nThe evaluation of active perception abilities on our benchmark, including zooming, and shifting. \"Model AVG\": average scores of column \"Zooming\", \"Shifting-R\", \"Shifting-E\", \"Shifting-M\", and \"Shifting-H\". \n\n#### Proprietary Models\n\n| Models            | Full Image | Zooming | Single View | Shifting-R | Shifting-E | Shifting-M | Shifting-H | Model AVG |\n|-------------------|------------|---------|-------------|------------|------------|------------|------------|-----------|\n| Gemini-1.5-pro | **73.85**  | **72.31**| 58.15       | **67.08**  | **67.38**  | **65.54**  | **67.69**  | **68.00** |\n| GPT-4o            | 67.38      | 68.62   | **61.23**   | 67.08      | 66.77      | 65.23      | 64.31      | 66.40     |\n| Claude 3.5 Sonnet | 72.92      | 71.69   | 54.46       | 65.23      | 66.15      | 60.31      | 61.85      | 65.05     |\n\n#### Open-source Models for Multiple Images as Input\n\n| Models                  | Full Image | Zooming | Single View | Shifting-R | Shifting-E | Shifting-M | Shifting-H | Model AVG |\n|-------------------------|------------|---------|-------------|------------|------------|------------|------------|-----------|\n| Qwen2-VL                | 63.08      | 64.62   | 54.46       | 61.23      |  **62.77** |  **64.31** |  **61.85** | **62.96** |\n| Idefics3-8B-Llama3       | 59.08      | 58.15   | 53.23       | **61.85** | 59.38      | 59.69      | 60.31      | 59.88     |\n| MiniCPM-V 2.6           | 64.62      | 61.85   | 54.46       | 54.77      | 61.23      | 58.15      | 55.69      | 58.34     |\n| mPLUG-Owl3              | 62.46      | 60.92   | 54.15       | 51.69      | 56.31      | 55.69      | 53.54      | 55.63     |\n| LLaVA-OneVision         | **64.92** | **65.23**|  **56.92**  | 53.54      | 57.23      | 52.31      | 48.62      | 55.39     |\n| InternVL2-8B            | 58.15      | 56.00   | 45.85       | 54.77      | 59.70      | 53.23      | 52.00      | 55.14     |\n| Mantis                  | 59.08      | 60.62   | 52.92       | 52.92      | 55.38      | 52.92      | 52.31      | 54.83     |\n| Idefics2-8B             | 61.85      | 61.85   | 55.69       | 53.23      | 56.92      | 51.69      | 49.23      | 54.58     |\n| Brote-IM-XL-3B          | 54.77      | 54.46   | 55.69       | 51.38      | 51.08      | 52.62      | 47.69      | 51.45     |\n| Idefics2-8B-base        | 52.62      | 48.62   | 47.69       | 49.54      | 50.77      | 47.69      | 47.69      | 48.86     |\n| Brote-IM-XXL-11B        | 53.85      | 54.77   | 49.23       | 49.85      | 50.77      | 44.92      | 43.69      | 48.80     |\n| MMICL-XXL-11B           | 51.69      | 49.54   | 50.15       | 49.85      | 49.85      | 46.77      | 45.54      | 48.31     |\n| MMICL-XL-3B             | 49.85      | 49.85   | 44.31       | 44.92      | 48.92      | 45.85      | 44.31      | 46.77     |\n\n#### Open-source Models for Single Image as Input\n\n| Models                  | Full Image | Zooming | Single View | Shifting-R | Shifting-E | Shifting-M | Shifting-H | Model AVG |\n|-------------------------|------------|---------|-------------|------------|------------|------------|------------|-----------|\n| MiniCPM-Llama3-V-2.5    | 63.87      | 61.25   | **54.47**| **60.92**| 60.31      | **59.38**| **58.46**| **60.06**|\n| GLM-4V-9B               | **67.08**| 56.92   | 53.85       | 56.92      | **60.62**| 56.00      | 52.92      | 56.68     |\n| InternVL-Vicuna-13B     | 56.92      | 62.77   | 52.31       | 53.85      | 52.92      | 52.92      | 51.08      | 54.71     |\n| LLaVA-1.6 7B            | 55.08      | **68.92**| 50.15       | 51.69      | 52.31      | 49.23      | 48.00      | 54.03     |\n| InternVL-Vicuna-7B      | 55.38      | 65.23   | 51.70       | 52.92      | 51.38      | 50.77      | 48.62      | 53.78     |\n| LLaVA-1.6 13B           | 56.92      | 65.23   | 52.31       | 45.85      | 55.08      | 52.62      | 48.92      | 53.54     |\n| InternVL-Vicuna-13B-448px| 50.46      | 57.85   | 45.54       | 48.31      | 48.31      | 48.92      | 48.92      | 50.46     |\n| mPLUG-Owl2-7B           | 55.08      | 55.38   | 52.00       | 47.38      | 46.46      | 46.46      | 46.15      | 48.37     |\n| Mini-Gemini-7B-HD       | 55.69      | 34.77   | 51.70       | 48.62      | 48.00      | 47.69      | 50.15      | 45.85     |\n| SEAL                    | 48.31      | 54.77   | 42.77       | 42.15      | 42.77      | 40.02      | 40.62      | 44.07     |\n| Mini-Gemini-7B          | 47.08      | 17.85   | 47.38       | 39.38      | 38.15      | 38.15      | 36.00      | 33.91     |\n\n### Results of Mixed Pipeline for Multi-Image Models\n\n**ACC**: accuracy; **\\#zoom**: average zooming operations; **\\#shift**: average shifting operations; **\\#view**: average used views.\n\n|   Models   | ACC       | #zoom | #shift | #view | \n|------------|-----------|-------|--------|-------|\n|   GPT-4o   | **69.54** | 1.61  | 1.23   | 1.35  | \n|  Qwen2-VL  |   65.54   | 2.51  | 2.17   | 2.12  | \n|MiniCPM-V 2.6|  64.00   | 1.31  | 0.39   | 0.94  |\n| mPLUG-Owl3 |   59.69   | 2.59  | 1.49   | 1.43  |\n|  Idefics3  |   62.15   | 1.16  | 0.59   | 0.58  |\n\n## Evaluation\n\nTo run the evaluation scripts, please modify the mehtods for loading models, inference, and post processing of model generated answers. \n\nYou can also use our provided scripts for some models (including Qwen2-VL, mPLUG-Owl3, Idefics3, Mantis, Brote, MMICL. API-based models will be available soon.)\n\nPlease try:\n```\nbash eval_script/run_qwen2vl_mix.sh 0,1,2,3 ../asset path_to_qwen2vl \u003e qwen_mix.log 2\u003e\u00261 \u0026\n```\n\n🚧 _These are temporal scripts and will be updated soon. If you find the scripts does not working properly, please raise issues in this repo._\n\nSample outputs are provided in **results** dir.\n\n## Citation\nIf you find our project useful, please consider citing:\n```\n@misc{wang2024activiewevaluatingactiveperception,\n      title={ActiView: Evaluating Active Perception Ability for Multimodal Large Language Models}, \n      author={Ziyue Wang and Chi Chen and Fuwen Luo and Yurui Dong and Yuanchi Zhang and Yuzhuang Xu and Xiaolong Wang and Peng Li and Yang Liu},\n      year={2024},\n      eprint={2410.04659},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV},\n      url={https://arxiv.org/abs/2410.04659}, \n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthunlp-mt%2Factiview","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthunlp-mt%2Factiview","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthunlp-mt%2Factiview/lists"}