https://github.com/ucas-vg/Sambor

Sambor: Boosting Segment Anything Model Towards Open-Vocabulary Learning
https://github.com/ucas-vg/Sambor

object-detection open-vocabulary segment-anything vision-language

Last synced: 2 months ago
JSON representation

Sambor: Boosting Segment Anything Model Towards Open-Vocabulary Learning

• Host: GitHub
• URL: https://github.com/ucas-vg/Sambor
• Owner: ucas-vg
• Created: 2023-12-05T06:32:31.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2023-12-07T04:10:25.000Z (about 1 year ago)
• Last Synced: 2023-12-08T04:32:31.960Z (about 1 year ago)
• Topics: object-detection, open-vocabulary, segment-anything, vision-language
• Homepage: https://arxiv.org/abs/2312.03628
• Size: 274 KB
• Stars: 8
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

• Awesome-Segment-Anything - [code

README

        
# Sambor

> **[Boosting Segment Anything Model Towards Open-Vocabulary Learning](https://arxiv.org/abs/2312.03628)**


> Xumeng Han, Longhui Wei, Xuehui Yu, Zhiyang Dou, Xin He, Kuiran Wang, Zhenjun Han, Qi Tian


## Method

![method](imgs/overview.png)

 **Overall architecture of Sambor. (Left)** We construct a SideFormer to extract features from SAM and inject CLIP visual features to enhance semantic understanding. Building upon a two-stage detector, we devise an Open-set RPN that augments the vanilla RPN with open-set proposals generated by SAM. The language branch of CLIP encodes concepts in parallel, thereby empowering the detector with open-vocabulary recognition. **(Right)** The specific implementations of the extractor and injector in SideFormer.