{"id":50677979,"url":"https://github.com/NTU-AI4X/ConceptSeg-R1","last_synced_at":"2026-06-25T15:00:41.534Z","repository":{"id":358484014,"uuid":"1241263123","full_name":"NTU-AI4X/ConceptSeg-R1","owner":"NTU-AI4X","description":"Segment Any Concept via Meta-Reinforcement Learning","archived":false,"fork":false,"pushed_at":"2026-06-03T09:27:19.000Z","size":33580,"stargazers_count":235,"open_issues_count":0,"forks_count":19,"subscribers_count":13,"default_branch":"main","last_synced_at":"2026-06-13T14:28:09.160Z","etag":null,"topics":["concept-segmentation","generalized-concept-segmentation","image-segmentation","image-segmentation-pytorch","object-segmentation","unified-concept-segmentation","unified-image-segmentation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NTU-AI4X.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-17T06:50:55.000Z","updated_at":"2026-06-12T02:44:22.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/NTU-AI4X/ConceptSeg-R1","commit_stats":null,"previous_names":["ntu-ai4x/conceptseg-r1"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/NTU-AI4X/ConceptSeg-R1","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NTU-AI4X%2FConceptSeg-R1","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NTU-AI4X%2FConceptSeg-R1/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NTU-AI4X%2FConceptSeg-R1/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NTU-AI4X%2FConceptSeg-R1/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NTU-AI4X","download_url":"https://codeload.github.com/NTU-AI4X/ConceptSeg-R1/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NTU-AI4X%2FConceptSeg-R1/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34780126,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-25T02:00:05.521Z","response_time":101,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concept-segmentation","generalized-concept-segmentation","image-segmentation","image-segmentation-pytorch","object-segmentation","unified-concept-segmentation","unified-image-segmentation"],"created_at":"2026-06-08T17:00:22.110Z","updated_at":"2026-06-25T15:00:41.526Z","avatar_url":"https://github.com/NTU-AI4X.png","language":"Python","funding_links":[],"categories":["Paper List"],"sub_categories":["Follow-up Papers"],"readme":"\u003cdiv align=\"center\"\u003e\n\n\u003ch1\u003eConceptSeg-R1\u003c/h1\u003e\n\n**ConceptSeg-R1: Segment Any Concept via Meta-Reinforcement Learning**\n\n[![arXiv](https://img.shields.io/badge/arXiv-2026-b31b1b?style=flat-square\u0026logo=arxiv)](https://arxiv.org/pdf/2605.20385)\n[![Project Page](https://img.shields.io/badge/🌐%20Project-Page-blueviolet?style=flat-square)](https://ntu-ai4x.github.io/ConceptSeg-R1/)\n[![HuggingFace](https://img.shields.io/badge/🤗%20Model-7B%20Weights-ffd21e?style=flat-square)](https://huggingface.co/zhaoyuan666/ConceptSeg-R1-7B)\n[![Dataset](https://img.shields.io/badge/🤗%20Dataset-ConceptSeg--Benchmark-ffd21e?style=flat-square)](https://huggingface.co/datasets/zhaoyuan666/ConceptSeg-Benchmark)\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue?style=flat-square)](LICENSE)\n[![Stars](https://img.shields.io/github/stars/yuanzhao-CVLAB/ConceptSeg-R1?style=flat-square)](https://github.com/yuanzhao-CVLAB/ConceptSeg-R1/stargazers)\n\n\u003cp\u003e\n  \u003ca href=\"#introduction\"\u003eIntroduction\u003c/a\u003e •\n  \u003ca href=\"#get-started\"\u003eGet Started\u003c/a\u003e •\n  \u003ca href=\"#data\"\u003eData\u003c/a\u003e •\n  \u003ca href=\"#datasets--checkpoints\"\u003eCheckpoints\u003c/a\u003e\n\u003c/p\u003e\n\n\n\u003cimg src=\"./assets/Concept_Tree.png\" width=\"90%\"/\u003e\n\u003c/div\u003e\n\n## 🎬 Short Video\n\u003ca href=\"https://ntu-ai4x.github.io/ConceptSeg-R1/#Show\"\u003e\n  \u003cimg src=\"https://github.com/NTU-AI4X/NTU-AI4X.github.io/blob/main/ConceptSeg-R1/ConceptSeg-R1-video.jpg\" width=\"90%\"\u003e\n\u003c/a\u003e\n\n## 📰 News\n\n- **May 2026** — arXiv paper released 🎉\n\n## 🗺️ Roadmap\n\n| Status | Item |\n|:------:|------|\n| ✅ | arXiv paper |\n| ✅ | Training code |\n| ✅ | Testing code |\n| ✅ | CI-CD-CR datasets |\n| ✅ | ConceptSeg-R1 (7B weights) |\n| ⬜ | Release larger MLLM  weights, e.g.,  ConceptSeg-R1-32B，ConceptSeg-R1-72B|\n\n\n## Introduction\n\n\u003cdiv align=\"center\"\u003e\n\n### 🌍 As segmentation in computer vision shifts from objects to concepts, \n### 🚀 **ConceptSeg-R1 takes the first step toward segmenting any concept.**\n\n\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"./assets/Architecture.png\" width=\"100%\"/\u003e\n\u003c/div\u003e\n\n\u003cbr\u003e\n\n### Key Contributions\n- **🌳 From Objects to Concepts**  \n  We introduce a three-level concept hierarchy covering **CI**, **CD**, and **CR** concepts, pushing segmentation beyond category recognition.\n\n- **🔁 From Instance Solving to Rule Induction**  \n  Meta-GRPO enables the model to infer transferable task rules from visual demonstrations and apply them deductively to unseen queries.\n\n- **🔗 Latent Concept Tokens for Frozen SAM 3**  \n  We map MLLM reasoning states into implicit concept tokens in the SAM 3 prompt space, enabling reasoning-aware segmentation without fine-tuning SAM 3.\n\n- **⚡ From Heavy Reasoning to Adaptive Inference**  \n  The Shortcut Router dynamically balances SAM 3 efficiency and reasoning depth, enabling fast perception for simple cases and deeper reasoning for complex concepts.\n\n## Results\n\n### Concept Segmentation Benchmarks (CI / CD / CR)\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"./assets/tab1.png\" width=\"100%\"/\u003e\n\u003c/div\u003e\n\u003cbr\u003e\n\n### Cityscapes Performance (Zero-Shot)\n\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"./assets/tab2.png\" width=\"90%\"/\u003e\n\u003c/div\u003e\n\u003cbr\u003e\n\n### ReasonSeg Performance (Zero-Shot)\n\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"./assets/tab3.png\" width=\"60%\"/\u003e\n\u003c/div\u003e\n\n### Qualitative Comparison\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"./assets/fig4.png\" width=\"100%\"/\u003e\n\u003c/div\u003e\n\u003cbr\u003e\n\n### Concept Coexistence\n\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"./assets/fig5.png\" width=\"100%\"/\u003e\n\u003c/div\u003e\n\u003cbr\u003e\n\n## Get Started\n\n\u003cdetails\u003e\n\u003csummary\u003e 1. Environment Setup\u003c/summary\u003e\n  \n### 1. Environment Setup\n\nBefore running `setup.sh`, download the release assets below from\n[GitHub Releases](https://github.com/yuanzhao-CVLAB/ConceptSeg-R1/releases)\nand place them in the repository root:\n\n- `sam3-main.zip`: the modified SAM 3 package used by ConceptSeg-R1.\n- `all_meta.json.zip`: the training metadata file.\n\n```bash\nconda create -n conceptseg-r1 python=3.10\nconda activate conceptseg-r1\nbash setup.sh\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e 2. Training \u003c/summary\u003e\n\n### 2. Training\n\n**Prepare data** — Download the dataset, extract `all_meta.json` through `setup.sh`,\nand set your `image_folders` path in the shell scripts.\n\n```bash\n# Stage 1: SFT Training\nbash run_grpo_multiimage_stage1.sh\n\n# Stage 2: GRPO Training\n# Note: Set `model_path` to the Stage 1 output checkpoint before running. （If you training encounter unexpected GPU OOM   despite sufficient VRAM,  try changing transformers_version to \"4.49.0\" in model_path/generation_config.json.）\nbash run_grpo_multiimage_stage2.sh\n```\n\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e 3. Evaluation \u003c/summary\u003e\n\n\n### 3. Evaluation \n\n**Concept Segmentation** — Download weights, set the model path in `eval_conceptseg.sh`, then run:\n\n```bash\nbash eval_conceptseg.sh\n```\n\n\u003e **Tip:** Configure specific tasks for testing inside `eval_conceptseg.sh`.\n\n**Reasoning Segmentation** — Download weights, set the model path in `eval_reasonseg.sh`, then run:\n\n```bash\nbash eval_reasonseg.sh\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e 4. Inference \u003c/summary\u003e\n\n### 4. Inference\n**Quick Start**: The `inference.sh` script includes 4 test cases covering different usage scenarios.\n```bash\n# Test 4  cases\nbash run_scripts/inference.sh\n```\n**Single Example Inference** — For quick testing and demonstration, use the inference script:\n```bash\n# Or test a specific case\npython src/eval/inference_single_example.py \\\n    --model_path \"path/to/model\" \\\n    --infer_path \"path/to/image\" \\\n    --question \"concept description\" \\\n    --output_path \"output/path\"\n```\n\n**Supported Input Modes:**\n- **Single Image**: Basic concept segmentation with text prompt (set `--ref_path` and `--bbox` to empty)\n- **Multiple Images**: Reference-guided segmentation with visual reasoning (set `--ref_path)\n- **Bounding Boxes**: Precise reference region specification for complex concepts (set `--bbox)\n \n\n\u003c/details\u003e\n \n## Data\n\n`all_meta.json` is no longer tracked in this repository. Download\n`all_meta.json.zip` from\n[GitHub Releases](https://github.com/yuanzhao-CVLAB/ConceptSeg-R1/releases)\nand run `bash setup.sh` to extract it before training.\n\nPlace datasets under a shared root directory (`image_folders`):\n\n```\nroot/\n├── isic2018/\n├── rare/\n├── Breast_Tumor/\n├── transparent1024/\n├── MGrounding-630k/\n├── Polyp/\n├── Shadow_detection/\n├── MIG-Bench/\n├── coco2014_Living/\n├── CoSOD3k1024/\n├── ultra_rare/\n├── coco2014_Artifact/\n├── fewshot1000/\n├── DUTS/\n├── ESDIDefects/\n└── COD10K1024/\n```\n\n\n## Metric\n\nEvaluation uses the [PySegMetric_EvalToolkit](https://github.com/Xiaoqi-Zhao-DLUT/PySegMetric_EvalToolkit).\n\n\n## Datasets \u0026 Checkpoints\n\n| Resource | Link |\n|----------|------|\n| 📦 ConceptSeg-Benchmark Dataset | [Download on HuggingFace](https://huggingface.co/datasets/zhaoyuan666/ConceptSeg-Benchmark) |\n| 🤖 ConceptSeg-R1-7B Weights | [Download on HuggingFace](https://huggingface.co/zhaoyuan666/ConceptSeg-R1-7B) |\n\n## Acknowledgements\n\nWe reference the excellent open-source repos [SAM 3](https://github.com/facebookresearch/sam3), [VLM-R1](https://github.com/om-ai-lab/VLM-R1) and [LENS](https://github.com/hustvl/LENS). Thanks to their authors for the valuable contributions to the community.\n\n## Citation\nIf you find this work useful, please consider starring  ⭐ and citing the repo!\n\n\n```bibtex\n@misc{zhao2026conceptseg,\n      title={ConceptSeg-R1: Segment Any Concept via Meta-Reinforcement Learning}, \n      author={Yuan Zhao and Youwei Pang and Jiaming Zuo and Wei Ji and Kailai Zhou and Bin Fan and Yunkang Cao and Lihe Zhang and Xiaofeng Liu and Huchuan Lu and Weisi Lin and Dacheng Tao and Xiaoqi Zhao},\n      year={2026},\n      eprint={2605.20385},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV},\n      url={https://arxiv.org/abs/2605.20385}, \n}\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNTU-AI4X%2FConceptSeg-R1","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FNTU-AI4X%2FConceptSeg-R1","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNTU-AI4X%2FConceptSeg-R1/lists"}