{"id":28420027,"url":"https://github.com/aim-uofa/segagent","last_synced_at":"2026-01-28T02:52:27.487Z","repository":{"id":281967262,"uuid":"943801259","full_name":"aim-uofa/SegAgent","owner":"aim-uofa","description":"[CVPR2025] SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectories","archived":false,"fork":false,"pushed_at":"2025-03-12T04:01:31.000Z","size":48389,"stargazers_count":48,"open_issues_count":1,"forks_count":1,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-06-05T03:15:00.175Z","etag":null,"topics":["agent","mllms","segment-anything","vlms"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aim-uofa.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-06T09:37:09.000Z","updated_at":"2025-06-04T09:11:20.000Z","dependencies_parsed_at":"2025-03-12T05:28:27.666Z","dependency_job_id":null,"html_url":"https://github.com/aim-uofa/SegAgent","commit_stats":null,"previous_names":["aim-uofa/segagent"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/aim-uofa/SegAgent","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aim-uofa%2FSegAgent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aim-uofa%2FSegAgent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aim-uofa%2FSegAgent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aim-uofa%2FSegAgent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aim-uofa","download_url":"https://codeload.github.com/aim-uofa/SegAgent/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aim-uofa%2FSegAgent/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262484112,"owners_count":23318395,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent","mllms","segment-anything","vlms"],"created_at":"2025-06-04T23:40:34.657Z","updated_at":"2026-01-28T02:52:27.482Z","avatar_url":"https://github.com/aim-uofa.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# 🎯 SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectories\n\n[Muzhi Zhu](https://scholar.google.com/citations?user=064gBH4AAAAJ\u0026hl=en)\u003csup\u003e1,2\u003c/sup\u003e, \u0026nbsp;\nYuzhuo Tian\u003csup\u003e1\u003c/sup\u003e, \u0026nbsp;\n[Hao Chen](https://stan-haochen.github.io/)\u003csup\u003e1*\u003c/sup\u003e, \u0026nbsp;\nChunluan Zhou\u003csup\u003e2\u003c/sup\u003e, \u0026nbsp;\nQingpei Guo\u003csup\u003e2*\u003c/sup\u003e, \u0026nbsp;\nYang Liu\u003csup\u003e1\u003c/sup\u003e, \u0026nbsp;\nMing Yang\u003csup\u003e2\u003c/sup\u003e, \u0026nbsp;\n[Chunhua Shen](https://cshen.github.io/)\u003csup\u003e1*\u003c/sup\u003e\n\n\u003csup\u003e1\u003c/sup\u003e[Zhejiang University](https://www.zju.edu.cn/english/), \u0026nbsp;\n\u003csup\u003e2\u003c/sup\u003e[Ant Group](https://www.antgroup.com/en)\n\n**CVPR2025**\n\n[📄 **Paper**](https://arxiv.org/abs/2503.08625)\u0026nbsp; | \u0026nbsp;[🌐 **Project Page**](https://aim-uofa.github.io/SegAgent/)\u0026nbsp; | \u0026nbsp;[🤖 **Model Weight**](https://www.modelscope.cn/models/zzzmmz/SegAgent-Model)\u0026nbsp; | \u0026nbsp;[📊 **Data**](https://www.modelscope.cn/models/zzzmmz/SegAgent-Dataset)\n\u003c/div\u003e\n\n## 🚀 Overview\n\u003cdiv align=\"center\"\u003e\n\u003cimg width=\"800\" alt=\"SegAgent Framework\" src=\"images/framework.png\"\u003e\n\u003c/div\u003e\n\n## 📖 Description\n\nMultimodal Large Language Models (MLLMs) demonstrate remarkable capabilities in understanding images but still struggle with pixel-level tasks like segmentation. SegAgent addresses this by introducing a novel **Human-Like Mask Annotation Task (HLMAT)**, enabling MLLMs to mimic the annotation trajectories of human experts using interactive segmentation tools.\n\nSegAgent effectively leverages these annotation trajectories without requiring architectural modifications or additional implicit tokens. Our approach significantly enhances MLLMs' segmentation and mask refinement abilities, establishing a new paradigm for assessing fine-grained visual understanding and multi-step reasoning.\n\n\n\n\n\n\n\n## 🚩 Plan\n- ✅ Release the weights.\n- ✅ Release the inference code.\n- ✅ Release the trajectory data for training and evaluation.\n\n\n\n## 🚀 Getting Started\n```bash\npip install -r  env.txt\n```\n\n## 🤖 Inference\n\nYou can run inference on the validation or test set using the trained model and the provided script:\n\n```bash\nbash run_eval.sh /path/to/your/trained_model\n```\n\nThis will run inference with **SimpleClick** as the segmentation model and **SegAgent** as the language grounding model. The script processes images and saves the predictions to the output directory.\n\nTo evaluate the results, run:\n\n```bash\npython eval_result_iou.py --input_json ./results/refcoco+_val_predictions.json\n```\n\n📄 For more details, refer to [`./evaltools/eval.md`](./evaltools/eval.md).\n\n---\n\n## 🧑‍🏫 Training\n\nSegAgent is trained using **Human-Like Mask Annotation Trajectories (HLMAT)**. Follow the steps below to launch the training process:\n\n### Step 1: Prepare the Data\n\nEnsure that the annotation trajectory data is preprocessed and saved in the appropriate format (e.g., COCO-style JSON files + click sequences).\n\nWe have uploaded the preprocessed trajectory data here:  \n📁 [SegAgent-Data](https://www.modelscope.cn/models/zzzmmz/SegAgent-Dataset)\n\nExample structure:\n\n```bash\ntree ./data/segagent-data\n├── refcoco_train.json\n├── refcoco_val.json\n├── refcoco+_train.json\n├── ...\n```\n\nAdditional image data sources:\n- RefCOCO image datasets: [LISA GitHub Repository](https://github.com/dvlab-research/LISA)\n- HQ segmentation (SAM-HQ): [Hugging Face SAM-HQ Data](https://huggingface.co/sam-hq-team/sam-hq-training/tree/main/data)\n\n### Step 2: Run Training\n\nWe recommend converting the trajectory data into a format supported by [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory), and training using their framework directly.\n\n---\n\n\n\n## 🎫 License\n\nFor academic usage, this project is licensed under [the 2-clause BSD License](LICENSE). For commercial inquiries, please contact [Chunhua Shen](mailto:chhshen@gmail.com).\n\n## 🖊️ Citation\n\nIf you find this work helpful for your research, please cite:\n\n```BibTeX\n@article{zhu2025segagent,\n  title={SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectories},\n  author={Zhu, Muzhi and Tian, Yuzhuo and Chen, Hao and Zhou, Chunluan and Guo, Qingpei and Liu, Yang and Yang, Ming and Shen, Chunhua},\n  journal={arXiv preprint arXiv:2503.08625},\n  year={2025},\n  url={https://arxiv.org/abs/2503.08625}\n}","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faim-uofa%2Fsegagent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faim-uofa%2Fsegagent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faim-uofa%2Fsegagent/lists"}