{"id":28403028,"url":"https://github.com/alpha-innovator/mme-reasoning","last_synced_at":"2025-06-26T17:31:50.264Z","repository":{"id":294931531,"uuid":"988371595","full_name":"Alpha-Innovator/MME-Reasoning","owner":"Alpha-Innovator","description":"Official Repository: A Comprehensive Benchmark for Logical Reasoning in MLLMs","archived":false,"fork":false,"pushed_at":"2025-06-17T13:08:14.000Z","size":4649,"stargazers_count":37,"open_issues_count":1,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-17T14:24:23.276Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://alpha-innovator.github.io/mmereasoning.github.io/#leaderboard","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Alpha-Innovator.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-22T12:58:21.000Z","updated_at":"2025-06-17T13:08:18.000Z","dependencies_parsed_at":"2025-06-17T14:32:34.143Z","dependency_job_id":null,"html_url":"https://github.com/Alpha-Innovator/MME-Reasoning","commit_stats":null,"previous_names":["alpha-innovator/mme-reasoning"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Alpha-Innovator/MME-Reasoning","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alpha-Innovator%2FMME-Reasoning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alpha-Innovator%2FMME-Reasoning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alpha-Innovator%2FMME-Reasoning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alpha-Innovator%2FMME-Reasoning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Alpha-Innovator","download_url":"https://codeload.github.com/Alpha-Innovator/MME-Reasoning/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alpha-Innovator%2FMME-Reasoning/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262113295,"owners_count":23261000,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-01T16:36:33.010Z","updated_at":"2025-06-26T17:31:50.242Z","avatar_url":"https://github.com/Alpha-Innovator.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MME-Reasoning 🔥: A Comprehensive Benchmark for Logical Reasoning in MLLMs\n\n![Multimodal Reasoning](https://img.shields.io/badge/Task-Multimodal_Reasoning-red) \n![Visual Reasoning](https://img.shields.io/badge/Task-Visual_Reasoning-red) \n![MME-Reasoning](https://img.shields.io/badge/Dataset-MME--Reasoning-blue) \n\n![OpenAI o4-mini](https://img.shields.io/badge/Model-OpenAI_o4--mini-green)\n![Seed1.5-VL-Thinking](https://img.shields.io/badge/Model-Seed1.5--VL--Thinking-green)\n![Gemini2.5-Pro-Thinking](https://img.shields.io/badge/Model-Gemini2.5--Pro--Thinking-green) \n\nOfficial repository for \"[MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMs]()\".\n\n🌟 For more details, please refer to the project page.\n\n[[🚀Project Page](https://alpha-innovator.github.io/mmereasoning.github.io/)] [[📖 Paper](https://arxiv.org/pdf/2505.21327)] [[📊 Huggingface Dataset](https://huggingface.co/datasets/U4R/MME-Reasoning)] [[🏆 Leaderboard](https://alpha-innovator.github.io/mmereasoning.github.io/#leaderboard)]\n\n## 💥 News\n- **[2025.06.17]** 🔥 We have integrated MME-Reasoning in [VLMEvalkit](https://github.com/open-compass/VLMEvalKit).\n- **[2025.05.23]** 🔥 We launch MME-Reasoning, a comprehensive benchmark designed to evaluate the reasoning ability of MLLMs. We release the [arxiv paper](https://arxiv.org/pdf/2505.21327) and all data samples in [huggingface dataset](https://huggingface.co/datasets/U4R/MME-Reasoning).\n\n## 👀 About MME-Reasoning\n\nLogical reasoning is a fundamental aspect of human intelligence and an essential capability for multimodal large language models (MLLMs). Existing benchmarks fail to comprehensively evaluate MLLMs reasoning abilities due to the lack of explicit categorization for logical reasoning types and an unclear understanding of reasoning.\n\nIn this paper, we introduce **MME-Reasoning**, a comprehensive benchmark specifically designed to evaluate the reasoning capability of MLLMs. MME-Reasoning consists of 1,188 carefully curated questions that systematically cover types of logical reasoning (**inductive**, **deductive**, and **abductive**), while spanning a range of difficulty levels.\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"assets/teaser.png\" width=\"70%\"\u003e \u003cbr\u003e\n\u003c/p\u003e\n\nExperiments were conducted on state-of-the-art MLLMs, covering Chat and Thinking types of both open-source and closed-source. Evaluations with MME-Reasoning reveal these key findings: **(1) MLLMs exhibit significant limitations and pronounced imbalances in reasoning capabilities.** **(2) Abductive reasoning remains a major bottleneck for current MLLMs.** **(3) Reasoning length scales with task difficulty, benefiting performance but accompanied by marginal effects and decreasing token efficiency.** We hope MME-Reasoning serves as a foundation for advancing multimodal reasoning in MLLMs.\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"assets/performance.png\" width=\"95%\"\u003e \u003cbr\u003e\n\u003c/p\u003e\n\n## Inference using [VLMEvalkit](https://github.com/open-compass/VLMEvalKit)\n\nPlease first install VLMEvalKit following the official github [repo](https://github.com/open-compass/VLMEvalKit).\n\nThen, run:\n\n```shell\npython run.py --data MME-Reasonig --model TESTED_MODEL --verbose\n```\n    \n## Inference\n\nWe are working to integrate the MME-Reasoning into existing VLMs evaluation frameworks. For the current version of the evaluation, please following the follows steps:\n\n1. Setup your environment following [VLMEvalKit](./README_VLMEVAL.md)\n2. Download MME-Reasoning data and metadata from [huggingface]().\n3. Set environment variable `LMUData` (note the images should exist under `$LMUDATA/MMEReasoning/images/`)\n4. Set the metadata path in `vlmeval/dataset/mmereasoning/mmereasoning.py` in `line 19` and `line 25`.\n5. Run:\n     ```python\n     python run.py --data MMEReasoning --model your_model --mode infer --verbose\n     ```\n6. Extract and judge the final results:\n     ```python\n     python test_mme_reasoning.py --file_path response_file\n     ```\n     The response file exists in outputs dir and ends with scores.xlsx.\n\n## 🏆 Leaderboard\n\n### Contributing to the Leaderboard\n\n🚀 The [Leaderboard](https://alpha-innovator.github.io/mmereasoning.github.io/#leaderboard) is continuously being updated, welcoming the contribution of your excellent MLLMs!\n\nTo contribute your model to the leaderboard, please email the prediction files to 📧[jkyuan112@gmail.com](mailto:jkyuan112@gmail.com) or [pengts521@gmail.com](mailto:pengts521@gmail.com).\n\n\n## :white_check_mark: Citation\n\nIf you find **MME-Reasoning** useful for your research and applications, please kindly cite using this BibTeX:\n\n```latex\n@article{yuan2025mme,\n  title={MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMs},\n  author={Yuan, Jiakang and Peng, Tianshuo and Jiang, Yilei and Lu, Yiting and Zhang, Renrui and Feng, Kaituo and Fu, Chaoyou and Chen, Tao and Bai, Lei and Zhang, Bo and others},\n  journal={arXiv preprint arXiv:2505.21327},\n  year={2025}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falpha-innovator%2Fmme-reasoning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falpha-innovator%2Fmme-reasoning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falpha-innovator%2Fmme-reasoning/lists"}