https://github.com/isbrycee/GEM
GEM: Boost Simple Network for Glass Surface Segmentation via Vision Foundation Models
https://github.com/isbrycee/GEM
glass-segmentation semantic-segmentation
Last synced: about 1 month ago
JSON representation
GEM: Boost Simple Network for Glass Surface Segmentation via Vision Foundation Models
- Host: GitHub
- URL: https://github.com/isbrycee/GEM
- Owner: isbrycee
- Created: 2023-07-28T08:35:37.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-05-22T02:25:45.000Z (12 months ago)
- Last Synced: 2024-05-22T07:29:40.824Z (12 months ago)
- Topics: glass-segmentation, semantic-segmentation
- Language: Python
- Homepage:
- Size: 22.2 MB
- Stars: 7
- Watchers: 1
- Forks: 1
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesome-diffusion-categorized - [Code
README
# .png)
GEM: Glass-Segmentor
[Jing Hao](https://scholar.google.com/citations?user=E8R8c00AAAAJ&hl=zh-CN), [Moyun Liu](https://scholar.google.com/citations?user=5gWDxqAAAAAJ&hl=zh-CN), [Jinrong Yang](https://scholar.google.com/citations?hl=zh-CN&user=8Of_NYQAAAAJ), [Kuo Feng Hung](https://scholar.google.com/citations?user=17V5x14AAAAJ&hl=zh-CN).
This repository is the official implementation of the [GEM: Boost Simple Network for Glass Surface Segmentation via Vision Foundation Models](https://arxiv.org/pdf/2307.12018). Our code is based on [MaskDINO](https://github.com/IDEA-Research/MaskDINO/tree/main).
Todo list
- [x] Release code and GEM-Tiny checkpoint
- [x] Release S-GSD dataset
- [x] Release multiple pre-trained models
- [x] Release GEM-Base model
***
### Features
* A simple but accurate segmentation framework, named **GEM**, for glass surface segmentation.
* Automatically construct a large-scale synthesized glass surface dataset with precise mask annotation, termed **S-GSD**.
* Surpasses the previous state-of-the-art methods by a large margin (IoU +2.1%).
## Installation
See [Mask DINO](https://github.com/IDEA-Research/MaskDINO/blob/main/INSTALL.md).
## Getting Started
See [Inference Demo with Pre-trained Model](demo/README.md)
See [Results](#results).
See [Preparing Datasets for GEM](datasets/README.md).
See [Getting Started](#getting-started-1).
***
# Results
Model
Pre-trained dataset
IoU
F_β
MAE
BER
FPS
download
GEM-Tiny | config
S-GSD-1x
0.755
0.852
0.038
8.39
16.09
BaiduDisk
GEM-Tiny | config
S-GSD-5x
0.757
0.855
0.035
8.54
16.09
BaiduDisk
GEM-Tiny | config
S-GSD-10x
0.764
0.866
0.034
8.62
16.09
BaiduDisk
GEM-Tiny | config
S-GSD-20x
0.770
0.865
0.032
8.21
16.09
BaiduDisk
GEM-Base | config
S-GSD-1x
0.766
0.873
0.031
9.44
11.55
BaiduDisk
GEM-Base | config
S-GSD-5x
0.769
0.858
0.032
8.16
11.55
BaiduDisk
GEM-Base | config
S-GSD-10x
0.774
0.868
0.032
8.56
11.55
BaiduDisk
GEM-Base | config
S-GSD-20x
0.774
0.865
0.029
8.35
11.55
BaiduDisk
**[07/15/2024] The sam backbone pre-trained model can download from [Huggingface]([https://huggingface.co/Bryceee/GEM-Glass-Segment-model/tree/main](https://huggingface.co/datasets/Gourieff/ReActor/blob/main/models/sams/sam_vit_b_01ec64.pth))**
**[07/15/2024] All pre-trained models can also download from [Huggingface](https://huggingface.co/Bryceee/GEM-Glass-Segment-model/tree/main)**
**[07/18/2024] S-GSD-1x dataset can also download from [Huggingface](https://huggingface.co/datasets/Bryceee/S-GSD)**
The download link of S-GSD-5x is blow here, it be divided into three parts:
Part1: https://pan.baidu.com/s/1CL0x8s1LXdIIoOjcr5wirw?pwd=2ff4
Part2: https://pan.baidu.com/s/18uCDKFmzy7vSmWm1IF4JFQ?pwd=9jqb
Part3: https://pan.baidu.com/s/1Z8pePl9Ps3QtrZB7WQ7aoQ?pwd=8fku
The S-GSD-10x and S-GSD-20x are not released because of the large disk storage, if you want to get these two large-scale datasets, feel free to contact me via [email protected].
***
***
# Getting Started
In the above tables, the corresponding model checkpoints can pre-trained models can be downloaded from the link.
If your dataset files are not under this repo, you need to add `export DETECTRON2_DATASETS=/path/to/your/data` or use Symbolic Link `ln -s`
to link the dataset into this repo before the following command first.
#### Evalaluate our pretrained models
* You can download our pretrained models and evaluate them with the following commands.
```sh
python train_net.py --eval-only --num-gpus 8 --config-file config_path MODEL.WEIGHTS /path/to/checkpoint_file
```
#### Train GEM to reproduce results
* Use the above command without `eval-only` will train the model. For MobileSAM/SAM backbones, you need to download its weight from () and specify the path of the pretrained backbones with `MODEL.WEIGHTS /path/to/pretrained_checkpoint`
```sh
python train_net.py --num-gpus 8 --config-file config_path MODEL.WEIGHTS /path/to/checkpoint_file
```
You can also refer to [Getting Started with Detectron2](https://github.com/facebookresearch/detectron2/blob/master/GETTING_STARTED.md) for full usage.
If you find our work helpful for your research, please consider citing the following BibTeX entry.
```BibTeX
@article{hao2024gem,
title={GEM: Boost Simple Network for Glass Surface Segmentation via Segment Anything Model and Data Synthesis},
author={Hao, Jing and Liu, Moyun and Hung, Kuo Feng},
journal={arXiv preprint arXiv:2401.15282},
year={2024}
}
```
If you find the code useful, please also consider the following BibTeX entry.
```BibTeX
@misc{li2022mask,
title={Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation},
author={Feng Li and Hao Zhang and Huaizhe xu and Shilong Liu and Lei Zhang and Lionel M. Ni and Heung-Yeung Shum},
year={2022},
eprint={2206.02777},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
```
## Acknowledgement
Many thanks to these excellent opensource projects
* [Mask DINO](https://github.com/IDEA-Research/MaskDINO)
* [Segment-Anything](https://github.com/facebookresearch/segment-anything)