{"id":28721349,"url":"https://github.com/visionxlab/space-10","last_synced_at":"2025-06-15T07:06:26.562Z","repository":{"id":298040394,"uuid":"996636741","full_name":"VisionXLab/SpaCE-10","owner":"VisionXLab","description":null,"archived":false,"fork":false,"pushed_at":"2025-06-11T12:04:17.000Z","size":1914,"stargazers_count":7,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-11T12:43:35.969Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/VisionXLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-05T08:33:18.000Z","updated_at":"2025-06-11T12:04:20.000Z","dependencies_parsed_at":"2025-06-11T12:43:37.671Z","dependency_job_id":null,"html_url":"https://github.com/VisionXLab/SpaCE-10","commit_stats":null,"previous_names":["cuzyoung/space-10","visionxlab/space-10"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/VisionXLab/SpaCE-10","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VisionXLab%2FSpaCE-10","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VisionXLab%2FSpaCE-10/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VisionXLab%2FSpaCE-10/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VisionXLab%2FSpaCE-10/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/VisionXLab","download_url":"https://codeload.github.com/VisionXLab/SpaCE-10/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VisionXLab%2FSpaCE-10/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259935602,"owners_count":22934387,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-15T07:06:25.992Z","updated_at":"2025-06-15T07:06:26.548Z","avatar_url":"https://github.com/VisionXLab.png","language":"Python","readme":"\n\n\u003cdiv align=\"center\"\u003e\n\n\u003ch1\u003e\u003cimg src=\"assets/space-10-logo.png\" width=\"8%\"\u003e SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligence\u003c/h1\u003e\n\n[Ziyang Gong](https://scholar.google.com/citations?user=cWip8QgAAAAJ\u0026hl=zh-CN\u0026oi=ao)\u003csup\u003e1*\u003c/sup\u003e,\n[Wenhao Li]()\u003csup\u003e2*\u003c/sup\u003e,\n[Oliver Ma]()\u003csup\u003e3\u003c/sup\u003e,\n[Songyuan Li](https://scholar.google.com/citations?user=dVQGfEEAAAAJ\u0026hl=zh-CN\u0026oi=ao)\u003csup\u003e4\u003c/sup\u003e,\n[Jiayi Ji](https://scholar.google.com/citations?user=xp_rICcAAAAJ\u0026hl=zh-CN\u0026oi=ao)\u003csup\u003e5\u003c/sup\u003e,\n[Xue Yang](https://scholar.google.com/citations?user=2xTlvV0AAAAJ\u0026hl=zh-CN)\u003csup\u003e1\u003c/sup\u003e,\n[Gen Luo](https://scholar.google.com/citations?user=EyZqU9gAAAAJ\u0026hl=zh-CN)\u003csup\u003e3\u003c/sup\u003e,\n[Junchi Yan]()\u003csup\u003e1\u003c/sup\u003e,\n[Rongrong Ji]()\u003csup\u003e2\u003c/sup\u003e\n\n\u003csup\u003e1\u003c/sup\u003e Shanghai Jiao Tong University, \n\u003csup\u003e2\u003c/sup\u003e Xiamen University,  \n\u003csup\u003e3\u003c/sup\u003e Shanghai AI Lab, \n\u003csup\u003e4\u003c/sup\u003e Sun Yat-sen University, \n\u003csup\u003e5\u003c/sup\u003e National University of Singapore\n\n\u003csup\u003e*\u003c/sup\u003e Equal contribution\n\n\u003cimg src=\"https://visitor-badge.laobi.icu/badge?page_id=Cuzyoung.SpaCE-10.readme\u0026left_color=lightgrey\u0026right_color=green\"\u003e\n\u003cimg src=\"https://img.shields.io/badge/Updating%3F-yes-brightgreen\"\u003e\n\u003cimg src=\"https://img.shields.io/github/stars/Cuzyoung/SpaCE-10.svg?logo=github\u0026label=Stars\u0026color=white\"\u003e\n\n\u003ca href=\"#\"\u003e\u003cimg src=\"https://img.shields.io/badge/🌐_Project_Page-blue?style=for-the-badge\"\u003e\u003c/a\u003e\n\u003ca href=\"https://huggingface.co/datasets/Cusyoung/SpaCE-10\"\u003e\u003cimg src=\"https://img.shields.io/badge/HuggingFace-Dataset-yellow?style=for-the-badge\u0026logo=huggingface\"\u003e\u003c/a\u003e\n\u003c/div\u003e\n\n\u003c!-- \u003ca href=\"#\"\u003e\u003cimg src=\"https://img.shields.io/badge/BaiduNetDisk-download-blue?style=for-the-badge\"\u003e\u003c/a\u003e --\u003e\n\u003c!-- \u003ca href=\"#\"\u003e\u003cimg src=\"https://img.shields.io/badge/ScanNet++-Scenes-green?style=for-the-badge\"\u003e\u003c/a\u003e --\u003e\n---\n# 🧠 What is SpaCE-10?\n\n**SpaCE-10** is a **compositional spatial intellegence benchmark** for evaluating **Multimodal Large Language Models (MLLMs)** in indoor environments. Our contribution as follows:\n\n- 🧬 We define an **Atomic Capability Pool**, proposing 10 **atomic spatial capabilities.**\n- 🔗 Based on the composition of different atomic capabilities, we design **8 compositional QA types**.\n- 📈 SpaCE-10 benchmark contains 5,000+ QA pairs.\n- 🏠 All QA pairs come from 811 indoor scenes (ScanNet++, ScanNet, 3RScan, ARKitScene)\n- 🌍 SpaCE-10 spans both 2D and 3D MLLM evaluations and can be seamlessly adapted to MLLMs that accept 3D scan input.\n\n\u003cdiv align=\"center\"\u003e\n\u003cbr\u003e\u003cbr\u003e\n\u003cimg src=\"assets/space-10-teaser.png\" width=\"100%\"\u003e\n\u003cbr\u003e\u003cbr\u003e\n\u003c/div\u003e\n\n\n---\n# 🔥🔥🔥 News\n\u003c!-- - 🤖 [2025/06/10] Baseline results with GPT-4o, InternVL, Claude-3, LLaVA and more are available. --\u003e\n- 🖼️ [2025/06/11] Scans for 3D MLLMs and our manually collected 3D snapshots will be coming soon.\n- 💻 [2025/06/10] Evaluation code is released at followings.\n- 📊 [2025/06/09] We have released the benchmark for 2D MLLMs at [Hugging Face](https://huggingface.co/datasets/Cusyoung/SpaCE-10).\n- 📚 [2025/06/09] The paper of SpaCE-10 is released at [Arxiv](https://arxiv.org/abs/2506.07966v1), and it will be updating continually!\n---\n\n\n\n# Environment \nThe evaluation of SpaCE-10 is based on lmms-eval. Thus, we follow the environment settings of lmms-eval.\n```bash\ngit clone https://github.com/Cuzyoung/SpaCE-10.git\ncd SpaCE-10\nuv venv dev --python=3.10\nsource dev/bin/activate\nuv pip install -e .\n```\n\n# Evaluation\nTake InternVL2.5-8B as an example:\n```bash\ncd lmms-eval/run_bash\nbash internvl2.5-8b.sh\n```\nNotably, each time we test a new model, the corresponding environment of this model needs to be installed.\n---\n\n# Citation\n@article{gong2025space10,\n  title={SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligence},\n  author={Ziyang Gong, Wenhao Li, Oliver Ma, Songyuan Li, Jiayi Ji, Xue Yang, Gen Luo, Junchi Yan, Rongrong Ji},\n  journal={arXiv preprint arXiv:2506.07966},\n  year={2025}\n}\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvisionxlab%2Fspace-10","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvisionxlab%2Fspace-10","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvisionxlab%2Fspace-10/lists"}