{"id":13441458,"url":"https://github.com/xinghaochen/TinySAM","last_synced_at":"2025-03-20T12:30:55.645Z","repository":{"id":213625833,"uuid":"733458537","full_name":"xinghaochen/TinySAM","owner":"xinghaochen","description":"Official PyTorch implementation of \"TinySAM: Pushing the Envelope for Efficient Segment Anything Model\"","archived":false,"fork":false,"pushed_at":"2024-04-01T14:14:43.000Z","size":13387,"stargazers_count":380,"open_issues_count":7,"forks_count":23,"subscribers_count":13,"default_branch":"main","last_synced_at":"2024-08-01T03:34:35.255Z","etag":null,"topics":["efficient-sam","sam","segment-anything","tinysam"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xinghaochen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-12-19T11:25:54.000Z","updated_at":"2024-07-31T14:29:54.000Z","dependencies_parsed_at":"2024-04-01T15:44:07.711Z","dependency_job_id":null,"html_url":"https://github.com/xinghaochen/TinySAM","commit_stats":null,"previous_names":["xinghaochen/tinysam"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xinghaochen%2FTinySAM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xinghaochen%2FTinySAM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xinghaochen%2FTinySAM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xinghaochen%2FTinySAM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xinghaochen","download_url":"https://codeload.github.com/xinghaochen/TinySAM/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221760002,"owners_count":16876332,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["efficient-sam","sam","segment-anything","tinysam"],"created_at":"2024-07-31T03:01:34.180Z","updated_at":"2025-03-20T12:30:55.640Z","avatar_url":"https://github.com/xinghaochen.png","language":"Python","funding_links":[],"categories":["Paper List","Python","⚡ Optimization \u0026 Mobile Deployment"],"sub_categories":["Follow-up Papers","🛠️ Optimization Resources"],"readme":"# TinySAM\n### **TinySAM: Pushing the Envelope for Efficient Segment Anything Model**\n\n*Han Shu, Wenshuo Li, Yehui Tang, Yiman Zhang, Yihao Chen, Houqiang Li, Yunhe Wang, Xinghao Chen*\n\n*AAAI 2025*\n\n\u003ca href=\"https://arxiv.org/abs/2312.13789\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=Paper\u0026message=arXiv\u0026color=red\u0026logo=arxiv\"\u003e\u003c/a\u003e\n\u003ca href=\"https://huggingface.co/spaces/merve/tinysam\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=HuggingFace\u0026message=Demo\u0026color=yellow\"\u003e\u003c/a\u003e\n[![Open in OpenXLab](https://cdn-static.openxlab.org.cn/app-center/openxlab_app.svg)](https://openxlab.org.cn/apps/detail/shuh15/TinySAM)\n\nhttps://github.com/user-attachments/assets/c4bb50ab-bc20-419d-8c22-e36fe6a661ee\n\n## Updates\n* **2025/01/09**: TinySAM is accepted by AAAI 2025.\n* **2024/01/06**: [Demo](https://openxlab.org.cn/apps/detail/shuh15/TinySAM) of TinySAM is now available in **OpenXLab**. Thanks for the GPU grant.\n* **2023/12/27**: [Models](https://huggingface.co/merve/tinysam) and [demo](https://huggingface.co/spaces/merve/tinysam) of TinySAM are now available in **Hugging Face**. Thanks for [merveenoyan](https://github.com/merveenoyan).\n* **2023/12/27**: Pre-trained models and codes of [Q-TinySAM](#usage) (quantized variant) are released.\n* **2023/12/27**: [Evaluation](#evaluation) codes for zero-shot instance segmentation task on COCO are released.\n* **2023/12/22**: Pre-trained models and codes of TinySAM are released both in [Pytorch](https://github.com/xinghaochen/TinySAM) and [Mindspore](https://gitee.com/mindspore/models/tree/master/research/cv/TinySAM).\n\n## Overview\n\nWe propose a framework to obtain a tiny segment anything model (**TinySAM**) while maintaining the strong zero-shot performance. We first propose a full-stage knowledge distillation method with online hard prompt sampling strategy to distill a lightweight student model. We also adapt the post-training quantization to the promptable segmentation task and further reducing the computational cost. Moreover, a hierarchical segmenting everything strategy is proposed to accelerate the everything inference by with almost no performance degradation. With all these proposed methods, our TinySAM leads to orders of magnitude computational reduction and pushes the envelope for efficient segment anything task. Extensive experiments on various zero-shot transfer tasks demonstrate the significantly advantageous performance of our TinySAM against counterpart methods.\n\n![framework](./fig/framework.png)\n\u003cdiv align=center\u003e\n\u003csup\u003eFigure 1: Overall framework and zero-shot results of TinySAM.\u003c/sup\u003e\n\u003c/div\u003e\n\n![everything](./fig/everything.png)\n\u003cdiv align=center\u003e\n\u003csup\u003eFigure 2: Our hierarchical strategy for everything mode.\u003c/sup\u003e\n\u003c/div\u003e\n\n![vis](./fig/vis.png)\n\u003cdiv align=center\u003e\n\u003csup\u003eFigure 3: Visualization results of TinySAM.\u003c/sup\u003e\n\u003c/div\u003e\n\n## Requirements\nThe code requires `python\u003e=3.7` and we use `torch==1.10.2` and `torchvision==0.11.3`. To visualize the results, `matplotlib\u003e=3.5.1` is also required.  \n- python 3.7\n- pytorch == 1.10.2\n- torchvision == 0.11.3\n- matplotlib==3.5.1\n\n## Usage\n\n1. Download [checkpoints](#evaluation) into the directory of *weights*.\n\n2. Run the demo code for single prompt of point or box.\n\n```\npython demo.py\n```\n3. Run the demo code for hierarchical segment everything strategy.\n```\npython demo_hierachical_everything.py\n```\n\n4. Run the demo code for quantization inference.\n```\npython demo_quant.py\n```\n\n## Evaluation\nWe follow the setting of original [SAM](https://arxiv.org/abs/2304.02643) paper and evaluate the zero-shot instance segmentaion on COCO and LVIS dataset. The experiment results are described as followed.\n\n| Model               | FLOPs (G) |COCO AP (%) | LVIS AP (%)| \n| ------------------- | -------- | ------- |------- |\n| SAM-H                 |2976| 46.6/46.5*     | 44.7       | \n| SAM-L                 |1491| 46.2/45.5*     | 43.5       | \n| SAM-B                 |487| 43.4/41.0*     | 40.8       | \n| FastSAM                 |344| 37.9     | 34.5       | \n| MobileSAM            | 42.0|41.0     | 37.0       | \n| **TinySAM**  [\\[ckpt\\]](https://github.com/xinghaochen/TinySAM/releases/download/3.0/tinysam_42.3.pth)       | 42.0|42.3     | 38.6       | \n| **Q-TinySAM**  [\\[ckpt\\]](https://github.com/xinghaochen/TinySAM/releases/download/2.0/tinysam_w8a8.pth)            | 20.3|41.4     | 37.2      | \n\n\u003csup\u003e\\* Results of single output ([`multimask_output=False`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/predictor.py#L98)). \u003c/sup\u003e\u003c/br\u003e\n\nFirst download the detection boxes ([`coco_instances_results_vitdet.json`](https://github.com/xinghaochen/TinySAM/releases/download/2.0/coco_instances_results_vitdet.json)) produced by ViTDet model, as well as the ground-truth instance segmentation labels([`instances_val2017.json`](https://github.com/xinghaochen/TinySAM/releases/download/2.0/instances_val2017.json)) and put them into `eval/json_files`. \nRelated json files for LVIS dataset are available in [`lvis_instances_results_vitdet.json`](https://github.com/xinghaochen/TinySAM/releases/download/3.0/lvis_instances_results_vitdet.json) and [`lvis_v1_val.json`](https://github.com/xinghaochen/TinySAM/releases/download/3.0/lvis_v1_val.json).\n\nRun the following code to perform evaluation for zero-shot instance segmentation on COCO dataset.\n```\ncd eval; sh eval_coco.sh\n```\n\n\n## Acknowledgements\nWe thank the following projects: [SAM](https://github.com/facebookresearch/segment-anything), [MobileSAM](https://github.com/ChaoningZhang/MobileSAM), [TinyViT](https://github.com/microsoft/Cream).\n\n## Citation\n```bibtex\n@article{tinysam,\n  title={TinySAM: Pushing the Envelope for Efficient Segment Anything Model},\n  author={Shu, Han and Li, Wenshuo and Tang, Yehui and Zhang, Yiman and Chen, Yihao and Li, Houqiang and Wang, Yunhe and Chen, Xinghao},\n  journal={arXiv preprint arXiv:2312.13789},\n  year={2023}\n}\n```\n\n## License\n\nThis project is licensed under \u003ca rel=\"license\" href=\"License.txt\"\u003e Apache License 2.0\u003c/a\u003e. Redistribution and use should follow this license.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxinghaochen%2FTinySAM","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxinghaochen%2FTinySAM","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxinghaochen%2FTinySAM/lists"}