https://github.com/yangr116/vst
Visual Spatial Tuning
https://github.com/yangr116/vst
spatial-intelligence spatial-reasoning spatial-understanding vlm
Last synced: 24 days ago
JSON representation
Visual Spatial Tuning
- Host: GitHub
- URL: https://github.com/yangr116/vst
- Owner: Yangr116
- License: apache-2.0
- Created: 2025-11-07T03:40:17.000Z (9 months ago)
- Default Branch: master
- Last Pushed: 2026-03-25T08:50:25.000Z (4 months ago)
- Last Synced: 2026-03-26T12:36:25.936Z (4 months ago)
- Topics: spatial-intelligence, spatial-reasoning, spatial-understanding, vlm
- Language: Jupyter Notebook
- Homepage: https://yangr116.github.io/vst_project/
- Size: 7.23 MB
- Stars: 189
- Watchers: 4
- Forks: 8
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
Visual Spatial Tuning
[](https://arxiv.org/abs/2511.05491)
[](https://yangr116.github.io/vst_project/)
[](https://huggingface.co/datasets/rayruiyang/vst_500k)
[](https://huggingface.co/collections/rayruiyang/vst)
We introduce **Visual Spatial Tuning (VST)**, a comprehensive framework designed to cultivate Vision-Language Models (VLMs) with human-like visuospatial abilities—from spatial perception to advanced reasoning.
---
## 🔥 News
* Support Qwen3VL training code, see [assets/train.md](./assets/train.md#qwen3vl) for more details.
* Training code has been updated and verified, please see [Train](###train), which is very efficient because of data packing.
## 💡 Key Highlights
* **VST-P**: 4.1M samples across 19 skills, spanning single images, multi-image scenarios, and videos—boosting spatial perception in VLMs.
* **VST-R**: 135K curated samples that teach models to reason in space, including step-by-step reasoning and rule-based data for reinforcement learning.
* **Progressive Training Pipeline**: Start with supervised fine-tuning to build foundational spatial perception, then reinforce spatial reasoning abilities via RL. VST achieves state-of-the-art results on spatial benchmarks (34.8% on MMSI-Bench, 61.2% on VSIBench) without compromising general capabilities.
* **Vision-Language-Action Models Enhanced**: The VST paradigm significantly strengthens robotic learning.
---
## 🏷️ Model Card
| Model Name | 🤗 HuggingFace |
|:-------------- |:----------------------:|
| VST-3B-SFT | [rayruiyang/VST-3B-SFT](https://huggingface.co/rayruiyang/VST-3B-SFT) |
| VST-3B-RL | [rayruiyang/VST-3B-RL](https://huggingface.co/rayruiyang/VST-3B-RL) |
| VST-7B-SFT | [rayruiyang/VST-7B-SFT](https://huggingface.co/rayruiyang/VST-7B-SFT) |
| VST-7B-RL | [rayruiyang/VST-7B-RL](https://huggingface.co/rayruiyang/VST-7B-RL) |
```shell
# download models into checkpoints
python tools/download_hf_model.py --model_list rayruiyang/VST-3B-SFT rayruiyang/VST-3B-RL rayruiyang/VST-7B-SFT rayruiyang/VST-7B-RL --local_dir checkpoints
```
Click to see performance 📈
📈 Spatial & General Benchmarks
ModelsCV3DSRMMSIBLINKVSIMMStarMMBRealworldQAMMMUOCRBAI2D
VST-3B-SFT84.454.130.259.157.958.080.968.445.283.782.5
VST-3B-RL84.256.531.357.257.758.980.568.549.880.982.4
VST-7B-SFT85.554.632.062.160.663.183.372.250.685.584.9
VST-7B-RL86.560.134.862.661.263.583.068.549.486.183.5
📈 VSIBench
MethodsAvg.Obj. CountAbs. Dist.Obj. SizeRoom SizeRel. DistRel. Dir.Route PlanAppr. Order
VST-3B-SFT57.969.345.471.862.459.046.038.770.2
VST-3B-RL57.766.645.072.860.959.947.640.768.3
VST-7B-SFT60.672.044.474.368.359.755.844.965.2
VST-7B-RL61.271.643.875.569.260.055.644.369.2
📈 SUN RGBD 3D Object Detection
MethodsAP@15
Seed1.5-VL33.5
Gemini-2.0-Pro32.5
Gemini Robotics-ER48.3
VST-3B-SFT37.3
VST-3B-RL40.1
VST-7B-SFT41.6
VST-7B-RL44.2
---
## ⚡ Getting Started
### Training & Evaluation
**SFT**: Please follow [assets/train.md](./assets/train.md) to prepare env, data and train models.
**RL**: Please follow [projects/spatial_rl/README.md](projects/spatial_rl/README.md) to prepare env, data and train models using RL.
**VLA**: Please follow [assets/vla.md](./assets/vla.md) to train vla models.
**Evaluation**: Please follow [benchmark/README.md](./benchmark/README.md) to evaluate models.
### Cookbook
| Cookbook | Description |
|---------------|-----------------------------------|
| [scene understanding](cookbook/scene_understanding.ipynb) | Example for single image and multi-image inference |
| [3d object detection](cookbook/object_detection_3d.ipynb) | Example for 3D object detection |
### Using 🤗 Transformers to Chat
Install the inference dependency:
```bash
pip install transformers==4.57.0
pip install qwen-vl-utils
```
Then:
```python
import torch
from transformers import Qwen2_5_VLForConditionalGeneration, AutoTokenizer, AutoProcessor
from qwen_vl_utils import process_vision_info
THINK_SYSTEM_PROMPT = "You are a helpful assistant. You should first think about the reasoning process in the mind and then provide the user with the answer. The reasoning process is enclosed within tags, i.e. reasoning process here answer here."
think_mesg = {
"role": "system",
"content": [{"type": "text", "text": THINK_SYSTEM_PROMPT}],
}
enable_thinking=False
model_path="rayruiyang/VST-7B-RL"
# We recommend enabling flash_attention_2 for better acceleration and memory saving, especially in multi-image and video scenarios.
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
model_path,
torch_dtype=torch.bfloat16,
attn_implementation="flash_attention_2",
device_map="auto",
)
# default processer
processor = AutoProcessor.from_pretrained(model_path, min_pixels = 256*28*28, max_pixels=1280*28*28)
messages = [
{
"role": "user",
"content": [
{
"type": "image",
"image": "http://images.cocodataset.org/train2017/000000075668.jpg",
},
{"type": "text", "text": "Consider the real-world 3D locations of the objects. Is the 'no motorcycle' sign directly above the red bus?"},
],
}
]
if enable_thinking:
messages.insert(0, think_mesg)
# Preparation for inference
text = processor.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
text=[text],
images=image_inputs,
videos=video_inputs,
padding=True,
return_tensors="pt",
)
inputs = inputs.to("cuda")
# Inference: Generation of the output
generated_ids = model.generate(**inputs, max_new_tokens=1280)
generated_ids_trimmed = [
out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text[0])
```
## 📊 Dataset Overview
### 🖼️ VST-Perception (VST-P)
- **4.1M samples** across **19 tasks** for supervised fine-tuning.
- Covers three primary vision scenarios: *single-image*, *multi-image*, and *video*.
- VLMs tuned on VST-P show strong improvements in spatial perception:
- ~20% boost on CVBench-3D
- ~5% increase on BLINK
- ~16% gain on VSIBench
### 🧠 VST-Reasoning (VST-R)
- **135K samples**, split into:
- **Reasoning steps (CoT)**: Teach models how to reason spatially.
- **Rule-checkable data**: Used in online RL to further enhance reasoning skills.
- VLMs tuned on VST-R demonstrate:
- 8.9% improvement on MMSI-Bench
---
There are 500K reproduced data points [[rayruiyang/vst_500k](https://huggingface.co/datasets/rayruiyang/vst_500k)] for academic purposes. You can download them:
```shell
python tools/download_hf_data.py --repo_id="rayruiyang/vst_500k" --local_dir $YOUR_LOCAL_PATH
```
> [!NOTE]
> * This data doesn't include the video files, please follow [here](./assets/train.md#prepare-vst-data) to prepare video files.
> * We use `<|image_pad|>` and `<|video_pad|>` as the image and video special token.
**[Optional]** You can parse the parquet data into a json file and raw images by:
```bash
python tools/parse_vst_500k.py --data_dir "$YOUR_LOCAL_PATH/vst_500k"
```
You will get the data:
```text
data/
├── images
├── vst_500k.json
```
## 📜 License
This project is licensed under the Apache License. See the [LICENSE](./LICENSE) file for details.
The VST-3B model is fine-tuned from Qwen2.5VL-3B, its license is [Qwen2.5VL-3B LICENSE](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct/blob/main/LICENSE).
## Acknowledgement
Thanks for the projects: [Qwen2.5VL](https://github.com/QwenLM/Qwen3-VL/tree/main), [VeOmni](https://github.com/ByteDance-Seed/VeOmni), [EasyR1](https://github.com/hiyouga/EasyR1), and [VLMEvalKit](https://github.com/open-compass/VLMEvalKit).
If you find VST useful for your research or applications, please ⭐ star the repo or cite our work:
```bibtex
@article{vst,
title={Visual Spatial Tuning},
author={Rui Yang, Ziyu Zhu, Yanwei Li, Jingjia Huang, Shen Yan, Siyuan Zhou, Zhe Liu, Xiangtai Li, Shuangye Li, Wenqian Wang, Yi Lin, Hengshuang Zhao},
journal={arXiv preprint arXiv:2511.05491},
year={2025}
}
```