Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/ziqi-jin/finetune-anything
Fine-tune SAM (Segment Anything Model) for computer vision tasks such as semantic segmentation, matting, detection ... in specific scenarios
https://github.com/ziqi-jin/finetune-anything
computer-vision deep-learning fine-tune segment-anything
Last synced: about 2 months ago
JSON representation
Fine-tune SAM (Segment Anything Model) for computer vision tasks such as semantic segmentation, matting, detection ... in specific scenarios
- Host: GitHub
- URL: https://github.com/ziqi-jin/finetune-anything
- Owner: ziqi-jin
- License: mit
- Created: 2023-04-06T08:37:16.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2023-08-05T13:33:13.000Z (over 1 year ago)
- Last Synced: 2024-08-03T23:24:09.994Z (5 months ago)
- Topics: computer-vision, deep-learning, fine-tune, segment-anything
- Language: Python
- Homepage:
- Size: 107 KB
- Stars: 720
- Watchers: 13
- Forks: 54
- Open Issues: 30
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Introduction
The [Segment Anything Model (SAM)](https://github.com/facebookresearch/segment-anything) has revolutionized computer vision. Relying on fine-tuning of SAM will solve a large number of basic computer vision tasks. We are designing a **class-aware one-stage** tool for training fine-tuning models based on SAM.
You need to supply the datasets for your tasks and the [supported task](#Supported-Tasks) name, this tool will help you to get a finetuned model for your task. You are also allowed to design your own extend-SAM model, and FA supply the training, testing and deploy process for you.
## Design
Finetune-Anything further encapsulates the three parts of the original SAM, i.e., Image Encoder Adapter, Prompt Encoder Adapter, and Mask Decoder Adatper. We will support the base extend-SAM model for each task. Users also could design your own customized modules in each adapter, use FA to design different adapters, and set whether the parameters of any module are fixed. For modules with unfixed parameters, parameters such as `lr`, `weight decay` can be set to coordinate with the fine-tuning of the model.
check details in [How_to_use](https://github.com/ziqi-jin/finetune-anything/blob/main/how_to_use_finetune_anything.md).
For example, MaskDecoder is encapsulated as MaskDecoderAdapter. The current MaskDecoderAdatper contains two parts, DecoderNeck and DecoderHead.
## Supported Tasks
- [x] Semantic Segmentation
- [x] train
- [x] eval
- [ ] test
- [ ] Matting
- [ ] Instance Segmentation
- [ ] Detection
## Supported Datasets
- [x] TorchVOCSegmentation
- [x] BaseSemantic
- [ ] BaseInstance
- [ ] BaseMatting
## Deploy
- [ ] Onnx export
## Support Plan
FA will be updated in the following order,
- Mattng (task)
- Prompt Part (structure)
- [MobileSAM](https://github.com/ChaoningZhang/MobileSAM) (model)
- Instance Segmentation (task)
# Usage
finetune-anything(FA) supports the entire training process of SAM model fine-tuning, including the modification of the model structure, as well as the model training, verification, and testing processes. For details, check the [How_to_use](https://github.com/ziqi-jin/finetune-anything/blob/main/how_to_use_finetune_anything.md), the [Quick Start](#Quick-Start) gives an example of quickly using FA to train a custom semantic segmentation model.
## Quick Start
### Install
- Step1
```
git clone https://github.com/ziqi-jin/finetune-anything.git
cd finetune-anything
pip install -r requirements.txt
```
- Step2
Download the SAM weights from [SAM repository](https://github.com/facebookresearch/segment-anything#model-checkpoints)
- Step3
Modify the contents of yaml file for the specific task in **/config**, e.g., ckpt_path, model_type ...
### Train
```
CUDA_VISIBLE_DEVICES=${your GPU number} python train.py --task_name semantic_seg
```
## One more thing
If you need to use loss, dataset, or other functions that are not supported by FA, please submit an issue, and I will help you to implement them. At the same time, developers are also welcome to develop new loss, dataset or other new functions for FA, please submit your PR (pull requests).
## Related Resources
- [Documents](https://github.com/ziqi-jin/finetune-anything/blob/main/how_to_use_finetune_anything.md)