{"id":31764387,"url":"https://github.com/yangr116/vst","last_synced_at":"2026-06-30T13:31:48.405Z","repository":{"id":324549779,"uuid":"1091461710","full_name":"Yangr116/VST","owner":"Yangr116","description":"Visual Spatial Tuning","archived":false,"fork":false,"pushed_at":"2026-03-25T08:50:25.000Z","size":7580,"stargazers_count":189,"open_issues_count":1,"forks_count":8,"subscribers_count":4,"default_branch":"master","last_synced_at":"2026-03-26T12:36:25.936Z","etag":null,"topics":["spatial-intelligence","spatial-reasoning","spatial-understanding","vlm"],"latest_commit_sha":null,"homepage":"https://yangr116.github.io/vst_project/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Yangr116.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-07T03:40:17.000Z","updated_at":"2026-03-25T08:50:28.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/Yangr116/VST","commit_stats":null,"previous_names":["yangr116/vst"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Yangr116/VST","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Yangr116%2FVST","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Yangr116%2FVST/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Yangr116%2FVST/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Yangr116%2FVST/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Yangr116","download_url":"https://codeload.github.com/Yangr116/VST/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Yangr116%2FVST/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34969682,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-30T02:00:05.919Z","response_time":92,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["spatial-intelligence","spatial-reasoning","spatial-understanding","vlm"],"created_at":"2025-10-09T23:51:21.534Z","updated_at":"2026-06-30T13:31:48.372Z","avatar_url":"https://github.com/Yangr116.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align='center'\u003e\n\u003ch1\u003eVisual Spatial Tuning\u003c/h1\u003e\n\n\n[![Paper](https://img.shields.io/badge/paper-5f16a8?style=for-the-badge\u0026logo=arxiv\u0026logoColor=white)](https://arxiv.org/abs/2511.05491)\n[![Project Page](https://img.shields.io/badge/Blog-3858bf?style=for-the-badge\u0026logo=homepage\u0026logoColor=white)](https://yangr116.github.io/vst_project/)\n[![Dataset](https://img.shields.io/badge/Datasets-4d8cd8?style=for-the-badge\u0026logo=huggingface\u0026logoColor=white)](https://huggingface.co/datasets/rayruiyang/vst_500k)\n[![Weights](https://img.shields.io/badge/Model%20Weights-63cad3?style=for-the-badge\u0026logo=huggingface\u0026logoColor=white)](https://huggingface.co/collections/rayruiyang/vst)\n\u003c/div\u003e\n\nWe introduce **Visual Spatial Tuning (VST)**, a comprehensive framework designed to cultivate Vision-Language Models (VLMs) with human-like visuospatial abilities—from spatial perception to advanced reasoning.\n\n![Teaser Image](assets/teaser.png)\n\n---\n\n## 🔥 News\n* Support Qwen3VL training code, see [assets/train.md](./assets/train.md#qwen3vl) for more details.\n* Training code has been updated and verified, please see [Train](###train), which is very efficient because of data packing.\n\n\n## 💡 Key Highlights\n\n* **VST-P**: 4.1M samples across 19 skills, spanning single images, multi-image scenarios, and videos—boosting spatial perception in VLMs.  \n* **VST-R**: 135K curated samples that teach models to reason in space, including step-by-step reasoning and rule-based data for reinforcement learning.  \n* **Progressive Training Pipeline**: Start with supervised fine-tuning to build foundational spatial perception, then reinforce spatial reasoning abilities via RL. VST achieves state-of-the-art results on spatial benchmarks (34.8% on MMSI-Bench, 61.2% on VSIBench) without compromising general capabilities.  \n* **Vision-Language-Action Models Enhanced**: The VST paradigm significantly strengthens robotic learning.\n\n---\n\n## 🏷️ Model Card\n\n| Model Name     | 🤗 HuggingFace |\n|:-------------- |:----------------------:|\n| VST-3B-SFT     | [rayruiyang/VST-3B-SFT](https://huggingface.co/rayruiyang/VST-3B-SFT)           |\n| VST-3B-RL      | [rayruiyang/VST-3B-RL](https://huggingface.co/rayruiyang/VST-3B-RL)           |\n| VST-7B-SFT     | [rayruiyang/VST-7B-SFT](https://huggingface.co/rayruiyang/VST-7B-SFT)           |\n| VST-7B-RL      | [rayruiyang/VST-7B-RL](https://huggingface.co/rayruiyang/VST-7B-RL)           |\n\n```shell\n# download models into checkpoints\npython tools/download_hf_model.py --model_list rayruiyang/VST-3B-SFT rayruiyang/VST-3B-RL rayruiyang/VST-7B-SFT rayruiyang/VST-7B-RL --local_dir checkpoints\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eClick to see performance 📈 \u003c/summary\u003e\n  \n  \u003ch3\u003e📈 Spatial \u0026 General Benchmarks\u003c/h3\u003e\n  \u003ctable\u003e\n    \u003ctr\u003e\n      \u003cth\u003eModels\u003c/th\u003e\u003cth\u003eCV\u003c/th\u003e\u003cth\u003e3DSR\u003c/th\u003e\u003cth\u003eMMSI\u003c/th\u003e\u003cth\u003eBLINK\u003c/th\u003e\u003cth\u003eVSI\u003c/th\u003e\u003cth\u003eMMStar\u003c/th\u003e\u003cth\u003eMMB\u003c/th\u003e\u003cth\u003eRealworldQA\u003c/th\u003e\u003cth\u003eMMMU\u003c/th\u003e\u003cth\u003eOCRB\u003c/th\u003e\u003cth\u003eAI2D\u003c/th\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003eVST-3B-SFT\u003c/td\u003e\u003ctd\u003e84.4\u003c/td\u003e\u003ctd\u003e54.1\u003c/td\u003e\u003ctd\u003e30.2\u003c/td\u003e\u003ctd\u003e59.1\u003c/td\u003e\u003ctd\u003e57.9\u003c/td\u003e\u003ctd\u003e58.0\u003c/td\u003e\u003ctd\u003e80.9\u003c/td\u003e\u003ctd\u003e68.4\u003c/td\u003e\u003ctd\u003e45.2\u003c/td\u003e\u003ctd\u003e83.7\u003c/td\u003e\u003ctd\u003e82.5\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003eVST-3B-RL\u003c/td\u003e\u003ctd\u003e84.2\u003c/td\u003e\u003ctd\u003e56.5\u003c/td\u003e\u003ctd\u003e31.3\u003c/td\u003e\u003ctd\u003e57.2\u003c/td\u003e\u003ctd\u003e57.7\u003c/td\u003e\u003ctd\u003e58.9\u003c/td\u003e\u003ctd\u003e80.5\u003c/td\u003e\u003ctd\u003e68.5\u003c/td\u003e\u003ctd\u003e49.8\u003c/td\u003e\u003ctd\u003e80.9\u003c/td\u003e\u003ctd\u003e82.4\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003eVST-7B-SFT\u003c/td\u003e\u003ctd\u003e85.5\u003c/td\u003e\u003ctd\u003e54.6\u003c/td\u003e\u003ctd\u003e32.0\u003c/td\u003e\u003ctd\u003e62.1\u003c/td\u003e\u003ctd\u003e60.6\u003c/td\u003e\u003ctd\u003e63.1\u003c/td\u003e\u003ctd\u003e83.3\u003c/td\u003e\u003ctd\u003e72.2\u003c/td\u003e\u003ctd\u003e50.6\u003c/td\u003e\u003ctd\u003e85.5\u003c/td\u003e\u003ctd\u003e84.9\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003eVST-7B-RL\u003c/td\u003e\u003ctd\u003e86.5\u003c/td\u003e\u003ctd\u003e60.1\u003c/td\u003e\u003ctd\u003e34.8\u003c/td\u003e\u003ctd\u003e62.6\u003c/td\u003e\u003ctd\u003e61.2\u003c/td\u003e\u003ctd\u003e63.5\u003c/td\u003e\u003ctd\u003e83.0\u003c/td\u003e\u003ctd\u003e68.5\u003c/td\u003e\u003ctd\u003e49.4\u003c/td\u003e\u003ctd\u003e86.1\u003c/td\u003e\u003ctd\u003e83.5\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/table\u003e\n\n  \u003ch3\u003e📈 VSIBench\u003c/h3\u003e\n  \u003ctable\u003e\n    \u003ctr\u003e\n      \u003cth\u003eMethods\u003c/th\u003e\u003cth\u003eAvg.\u003c/th\u003e\u003cth\u003eObj. Count\u003c/th\u003e\u003cth\u003eAbs. Dist.\u003c/th\u003e\u003cth\u003eObj. Size\u003c/th\u003e\u003cth\u003eRoom Size\u003c/th\u003e\u003cth\u003eRel. Dist\u003c/th\u003e\u003cth\u003eRel. Dir.\u003c/th\u003e\u003cth\u003eRoute Plan\u003c/th\u003e\u003cth\u003eAppr. Order\u003c/th\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003eVST-3B-SFT\u003c/td\u003e\u003ctd\u003e57.9\u003c/td\u003e\u003ctd\u003e69.3\u003c/td\u003e\u003ctd\u003e45.4\u003c/td\u003e\u003ctd\u003e71.8\u003c/td\u003e\u003ctd\u003e62.4\u003c/td\u003e\u003ctd\u003e59.0\u003c/td\u003e\u003ctd\u003e46.0\u003c/td\u003e\u003ctd\u003e38.7\u003c/td\u003e\u003ctd\u003e70.2\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003eVST-3B-RL\u003c/td\u003e\u003ctd\u003e57.7\u003c/td\u003e\u003ctd\u003e66.6\u003c/td\u003e\u003ctd\u003e45.0\u003c/td\u003e\u003ctd\u003e72.8\u003c/td\u003e\u003ctd\u003e60.9\u003c/td\u003e\u003ctd\u003e59.9\u003c/td\u003e\u003ctd\u003e47.6\u003c/td\u003e\u003ctd\u003e40.7\u003c/td\u003e\u003ctd\u003e68.3\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003eVST-7B-SFT\u003c/td\u003e\u003ctd\u003e60.6\u003c/td\u003e\u003ctd\u003e72.0\u003c/td\u003e\u003ctd\u003e44.4\u003c/td\u003e\u003ctd\u003e74.3\u003c/td\u003e\u003ctd\u003e68.3\u003c/td\u003e\u003ctd\u003e59.7\u003c/td\u003e\u003ctd\u003e55.8\u003c/td\u003e\u003ctd\u003e44.9\u003c/td\u003e\u003ctd\u003e65.2\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003eVST-7B-RL\u003c/td\u003e\u003ctd\u003e61.2\u003c/td\u003e\u003ctd\u003e71.6\u003c/td\u003e\u003ctd\u003e43.8\u003c/td\u003e\u003ctd\u003e75.5\u003c/td\u003e\u003ctd\u003e69.2\u003c/td\u003e\u003ctd\u003e60.0\u003c/td\u003e\u003ctd\u003e55.6\u003c/td\u003e\u003ctd\u003e44.3\u003c/td\u003e\u003ctd\u003e69.2\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/table\u003e\n\n  \u003ch3\u003e📈 SUN RGBD 3D Object Detection\u003c/h3\u003e\n  \u003ctable\u003e\n    \u003ctr\u003e\n      \u003cth\u003eMethods\u003c/th\u003e\u003cth\u003eAP@15\u003c/th\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003eSeed1.5-VL\u003c/td\u003e\u003ctd\u003e33.5\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003eGemini-2.0-Pro\u003c/td\u003e\u003ctd\u003e32.5\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003eGemini Robotics-ER\u003c/td\u003e\u003ctd\u003e\u003cb\u003e48.3\u003c/b\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003eVST-3B-SFT\u003c/td\u003e\u003ctd\u003e37.3\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003eVST-3B-RL\u003c/td\u003e\u003ctd\u003e40.1\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003eVST-7B-SFT\u003c/td\u003e\u003ctd\u003e41.6\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003eVST-7B-RL\u003c/td\u003e\u003ctd\u003e\u003cb\u003e44.2\u003c/b\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/table\u003e\n\u003c/details\u003e\n\n\n---\n\n\n\n## ⚡ Getting Started\n\n### Training \u0026 Evaluation\n\n**SFT**: Please follow [assets/train.md](./assets/train.md) to prepare env, data and train models.\n\n**RL**: Please follow [projects/spatial_rl/README.md](projects/spatial_rl/README.md) to prepare env, data and train models using RL.\n\n**VLA**: Please follow [assets/vla.md](./assets/vla.md) to train vla  models.\n\n**Evaluation**: Please follow [benchmark/README.md](./benchmark/README.md) to evaluate models.\n\n\n### Cookbook\n\n| Cookbook           | Description                       |\n|---------------|-----------------------------------|\n| [scene understanding](cookbook/scene_understanding.ipynb) | Example for single image and multi-image inference           |\n| [3d object detection](cookbook/object_detection_3d.ipynb) | Example for 3D object detection             |\n\n\n### Using 🤗  Transformers to Chat\n\nInstall the inference dependency:\n```bash\npip install transformers==4.57.0\npip install qwen-vl-utils\n```\n\nThen:\n```python\nimport torch\nfrom transformers import Qwen2_5_VLForConditionalGeneration, AutoTokenizer, AutoProcessor\nfrom qwen_vl_utils import process_vision_info\n\nTHINK_SYSTEM_PROMPT = \"You are a helpful assistant. You should first think about the reasoning process in the mind and then provide the user with the answer. The reasoning process is enclosed within \u003cthink\u003e \u003c/think\u003e tags, i.e. \u003cthink\u003e reasoning process here \u003c/think\u003e answer here.\"\nthink_mesg = {\n                \"role\": \"system\",\n                \"content\": [{\"type\": \"text\", \"text\": THINK_SYSTEM_PROMPT}],\n            }\n\nenable_thinking=False\n\nmodel_path=\"rayruiyang/VST-7B-RL\"\n\n# We recommend enabling flash_attention_2 for better acceleration and memory saving, especially in multi-image and video scenarios.\nmodel = Qwen2_5_VLForConditionalGeneration.from_pretrained(\n    model_path,\n    torch_dtype=torch.bfloat16,\n    attn_implementation=\"flash_attention_2\",\n    device_map=\"auto\",\n)\n\n# default processer\nprocessor = AutoProcessor.from_pretrained(model_path, min_pixels = 256*28*28, max_pixels=1280*28*28)\n\nmessages = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\n                \"type\": \"image\",\n                \"image\": \"http://images.cocodataset.org/train2017/000000075668.jpg\",\n            },\n            {\"type\": \"text\", \"text\": \"Consider the real-world 3D locations of the objects. Is the 'no motorcycle' sign directly above the red bus?\"},\n        ],\n    }\n]\n\nif enable_thinking:\n    messages.insert(0, think_mesg)\n\n\n# Preparation for inference\ntext = processor.apply_chat_template(\n    messages, tokenize=False, add_generation_prompt=True\n)\nimage_inputs, video_inputs = process_vision_info(messages)\ninputs = processor(\n    text=[text],\n    images=image_inputs,\n    videos=video_inputs,\n    padding=True,\n    return_tensors=\"pt\",\n)\ninputs = inputs.to(\"cuda\")\n\n# Inference: Generation of the output\ngenerated_ids = model.generate(**inputs, max_new_tokens=1280)\ngenerated_ids_trimmed = [\n    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)\n]\noutput_text = processor.batch_decode(\n    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False\n)\nprint(output_text[0])\n```\n\n\n## 📊 Dataset Overview\n\n![Dataset Image](assets/dataset.jpg)\n\n### 🖼️ VST-Perception (VST-P)\n- **4.1M samples** across **19 tasks** for supervised fine-tuning.\n- Covers three primary vision scenarios: *single-image*, *multi-image*, and *video*.\n- VLMs tuned on VST-P show strong improvements in spatial perception:\n  - ~20% boost on CVBench-3D\n  - ~5% increase on BLINK\n  - ~16% gain on VSIBench\n\n### 🧠 VST-Reasoning (VST-R)\n- **135K samples**, split into:\n  - **Reasoning steps (CoT)**: Teach models how to reason spatially.\n  - **Rule-checkable data**: Used in online RL to further enhance reasoning skills.\n- VLMs tuned on VST-R demonstrate:\n  - 8.9% improvement on MMSI-Bench\n\n---\n\nThere are 500K reproduced data points [[rayruiyang/vst_500k](https://huggingface.co/datasets/rayruiyang/vst_500k)] for academic purposes. You can download them:\n```shell\npython tools/download_hf_data.py --repo_id=\"rayruiyang/vst_500k\" --local_dir $YOUR_LOCAL_PATH\n```\n\u003e [!NOTE]\n\u003e *  This data doesn't include the video files, please follow [here](./assets/train.md#prepare-vst-data) to prepare video files.\n\u003e *  We use `\u003c|image_pad|\u003e` and `\u003c|video_pad|\u003e` as the image and video special token. \n\n**[Optional]** You can parse the parquet data into a json file and raw images by:\n```bash\npython tools/parse_vst_500k.py --data_dir \"$YOUR_LOCAL_PATH/vst_500k\"\n```\nYou will get the data:\n```text\ndata/\n├── images\n├── vst_500k.json\n```\n\n\n\n## 📜 License\nThis project is licensed under the Apache License. See the [LICENSE](./LICENSE) file for details.\n\nThe VST-3B model is fine-tuned from Qwen2.5VL-3B, its license is [Qwen2.5VL-3B LICENSE](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct/blob/main/LICENSE).\n\n## Acknowledgement\n\nThanks for the projects: [Qwen2.5VL](https://github.com/QwenLM/Qwen3-VL/tree/main), [VeOmni](https://github.com/ByteDance-Seed/VeOmni), [EasyR1](https://github.com/hiyouga/EasyR1), and [VLMEvalKit](https://github.com/open-compass/VLMEvalKit).\n\nIf you find VST useful for your research or applications, please ⭐ star the repo or cite our work:\n\n```bibtex\n@article{vst,\n  title={Visual Spatial Tuning},\n  author={Rui Yang, Ziyu Zhu, Yanwei Li, Jingjia Huang, Shen Yan, Siyuan Zhou, Zhe Liu, Xiangtai Li, Shuangye Li, Wenqian Wang, Yi Lin, Hengshuang Zhao},\n  journal={arXiv preprint arXiv:2511.05491},\n  year={2025}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyangr116%2Fvst","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyangr116%2Fvst","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyangr116%2Fvst/lists"}