{"id":22068797,"url":"https://github.com/czg1225/SlimSAM","last_synced_at":"2025-07-24T07:31:01.869Z","repository":{"id":212011565,"uuid":"721182288","full_name":"czg1225/SlimSAM","owner":"czg1225","description":"[NeurIPS 2024] SlimSAM: 0.1% Data Makes Segment Anything Slim","archived":false,"fork":false,"pushed_at":"2024-11-17T06:56:24.000Z","size":37685,"stargazers_count":303,"open_issues_count":14,"forks_count":17,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-11-17T07:29:51.834Z","etag":null,"topics":["knowledge-distillation","model-compression","model-pruning","segment-anything-model"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/czg1225.png","metadata":{"files":{"readme":"README.md","changelog":"changepth.py","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-11-20T14:22:50.000Z","updated_at":"2024-11-17T06:56:28.000Z","dependencies_parsed_at":"2024-03-18T14:40:33.776Z","dependency_job_id":null,"html_url":"https://github.com/czg1225/SlimSAM","commit_stats":null,"previous_names":["czg1225/slimsam"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/czg1225%2FSlimSAM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/czg1225%2FSlimSAM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/czg1225%2FSlimSAM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/czg1225%2FSlimSAM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/czg1225","download_url":"https://codeload.github.com/czg1225/SlimSAM/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227421331,"owners_count":17775010,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["knowledge-distillation","model-compression","model-pruning","segment-anything-model"],"created_at":"2024-11-30T20:04:21.802Z","updated_at":"2025-07-24T07:31:01.847Z","avatar_url":"https://github.com/czg1225.png","language":"Python","funding_links":[],"categories":["Paper List","Networks"],"sub_categories":["Follow-up Papers"],"readme":"\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"images/paper/title.PNG\" width=\"33%\"\u003e\n\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\n\u003ch1\u003eSlimSAM: 0.1% Data Makes Segment Anything Slim\u003c/h1\u003e\n  \u003cdiv align=\"center\"\u003e\n  \u003ca href=\"https://arxiv.org/abs/2312.05284\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Conference-NeurIPS'2024-1A63BD.svg?style=flat-square\" alt=\"BLOOM\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://pytorch.org/\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/PyTorch-%3E=v1.7.0-EE4C2C.svg?style=flat-square\" alt=\"PyTorch\u003e=v1.7.1\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://huggingface.co/Zigeng/SlimSAM-uniform-50\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/HuggingFace-SlimSAM50-FFB000.svg?style=flat-square\" alt=\"LLaMA\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://huggingface.co/Zigeng/SlimSAM-uniform-77\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/HuggingFace-SlimSAM77-FAB093.svg?style=flat-square\" alt=\"Llama-2\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://huggingface.co/spaces/Xenova/segment-anything-web\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Demo-Huggingface-924E7D.svg?style=flat-square\" alt=\"Vicuna\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://colab.research.google.com/drive/1AQBGqjI51IERVibBKigTz_sra3CIVgR4?usp=sharing\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Demo-Colab-924E7D.svg?style=flat-square\" alt=\"Vicuna\"\u003e\n  \u003c/a\u003e\n\u003c/div\u003e\n\u003c/div\u003e\n\n\n\u003e **0.1% Data Makes Segment Anything Slim**   \n\u003e [Zigeng Chen](https://github.com/czg1225), [Gongfan Fang](https://fangggf.github.io/), [Xinyin Ma](https://horseee.github.io/), [Xinchao Wang](https://sites.google.com/site/sitexinchaowang/)   \n\u003e [xML Lab](https://sites.google.com/view/xml-nus), National University of Singapore  \n\u003e Paper: [[Paper]](https://arxiv.org/abs/2312.05284)\n\u003c!-- \u003e Colab: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1AQBGqjI51IERVibBKigTz_sra3CIVgR4?usp=sharing) --\u003e\n\n\n### Updates\n* 🚀 **March 22, 2024**: Awesome-Efficient-Segment-Anything is now available. Find more efficient SAMs [here](https://github.com/czg1225/Awesome-Efficient-Segment-Anything).\n* 🚀 **January 10, 2024**: Run **SlimSAM** in your browser with :hugs: Transformers.js ([demo](https://huggingface.co/spaces/Xenova/segment-anything-web)).\n* 🚀 **January 9, 2024**: Quickly loading using huggingface :hugs: :hugs: :hugs: .\n* 🚀 **January 7, 2024**: Release models using uniform local pruning for easier state dict loading.\n* 🚀 **December 19, 2023**: Release the Colab example for **SlimSAM**.\n* 🚀 **December 11, 2023**: Release the training code, inference code and pre-trained models for **SlimSAM**. \n\n![everything](images/paper/everything2.PNG)\n\n## Fast Start 🚀\n\n#### Quickly loading with Huggingface :hugs::\n\n``` python\nfrom PIL import Image\nfrom transformers import SamModel, SamProcessor\n\nmodel = SamModel.from_pretrained(\"Zigeng/SlimSAM-uniform-50\").to(\"cuda\")\nprocessor = SamProcessor.from_pretrained(\"Zigeng/SlimSAM-uniform-50\")\n\nimg_url = \"https://huggingface.co/ybelkada/segment-anything/resolve/main/assets/car.png\"\nraw_image = Image.open(requests.get(img_url, stream=True).raw).convert(\"RGB\")\ninput_points = [[[450, 600]]] # 2D localization of a window\ninputs = processor(raw_image, input_points=input_points, return_tensors=\"pt\").to(\"cuda\")\noutputs = model(**inputs)\nmasks = processor.image_processor.post_process_masks(outputs.pred_masks.cpu(), inputs[\"original_sizes\"].cpu(), inputs[\"reshaped_input_sizes\"].cpu())\nscores = outputs.iou_scores\n```\n\n## Introduction\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"images/paper/intro.png\" width=\"80%\"\u003e\n\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"images/paper/process.png\" width=\"80%\"\u003e\n\u003c/div\u003e\n\n**SlimSAM** is a novel data-efficient SAM compression method that achieves superior performance with extremely less training data. The essence of SlimSAM is encapsulated in the alternate slimming framework which effectively enhances knowledge inheritance under severely limited training data availability and exceptional pruning ratio. Diverging from prior techniques, our framework progressively compresses the model by alternately pruning and distilling distinct, decoupled sub-structures. Disturbed Taylor pruning is also proposed to address the misalignment between the pruning objective and training target, thereby boosting the post-distillation after pruning.\n\n![SlimSAM process](images/paper/frame.png)\n\nSlimSAM yields significant performance improvements while demanding **over 10 times less** training data than any other existing compression methods. Even when compared to the original SAM, SlimSAM achieves approaching performance while reducing parameter counts to merely **1.4\\% (9.1M)**, MACs to **0.8\\% (23G)**, and requiring only **0.1\\% (10k)** of the SAM training data.\n\n## Visualization Results\n\nQualitative comparison of results obtained using point prompts, box prompts, and segment everything prompts are shown.\n### Box Prompts and Point Prompts\n\n![prompt](images/paper/prompt.PNG)\n\n## Quantitative Results\n\nWe conducted a comprehensive comparison encompassing performance, efficiency, and training costs with other SAM compression methods and structural pruning methods.\n\n### Comparing with other SAM compression methods.\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"images/paper/compare_tab1.PNG\" width=\"90%\"\u003e\n\u003c/div\u003e\n\n### Comparing with other structural pruning methods.\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"images/paper/compare_tab2.PNG\" width=\"70%\"\u003e\n\u003c/div\u003e\n\n## Installation\n\nThe code requires `python\u003e=3.8`, as well as `pytorch\u003e=1.7` and `torchvision\u003e=0.8`. Please follow the instructions [here](https://pytorch.org/get-started/locally/) to install both PyTorch and TorchVision dependencies. Installing both PyTorch and TorchVision with CUDA support is strongly recommended.\n\n\nInstall with\n\n```\npip install -e .\n```\n\nThe following optional dependencies are necessary for mask post-processing, saving masks in COCO format.\n\n```\npip install opencv-python pycocotools matplotlib \n```\n\n## Dataset\nWe use the original SA-1B dataset in our code. See [here](https://ai.facebook.com/datasets/segment-anything/) for an overview of the datastet. The dataset can be downloaded [here](https://ai.facebook.com/datasets/segment-anything-downloads/).\n\nThe download dataset should be saved as:\n\n```\n\u003ctrain_data_root\u003e/\n      sa_xxxxxxx.jpg\n      sa_xxxxxxx.json\n      ......\n\u003cval_data_root\u003e/\n      sa_xxxxxxx.jpg\n      sa_xxxxxxx.json\n      ......\n\n```\n\n\nTo decode a mask in COCO RLE format into binary:\n\n``` python\nfrom pycocotools import mask as mask_utils\nmask = mask_utils.decode(annotation[\"segmentation\"])\n```\n\nSee [here](https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocotools/mask.py) for more instructions to manipulate masks stored in RLE format.\n\n\n## \u003ca name=\"Models\"\u003e\u003c/a\u003eModel Checkpoints\n\nThe base model of our method is available. To enhance collaboration with our dependency dectection algorithm, we have split the original image encoder's qkv layer into three distinct linear layers: q, k, and v.\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"images/paper/split.PNG\" width=\"70%\"\u003e\n\u003c/div\u003e\n\n\n\nClick the links below to download the checkpoints of orginal SAM-B.\n\n- `SAM-B`: [SAM-B model.](https://drive.google.com/file/d/1CtcyOm4h9bXgBF8DEVWn3N7g9-3r4Xzz/view?usp=sharing)\n\nThe check points of our SlimSAM are avalable. We release two versions, which are SlimSAM-50 (pruning ratio = 50%) and SlimSAM-77 (pruning ratio = 77%).\n\nClick the links below to download the checkpoints for the corresponding pruning ratio.\n\n#### Global Pruning Models:\n\n- `SlimSAM-50`: [SlimSAM-50 model.](https://drive.google.com/file/d/1bTjBZs2oWHeo6OPxumD_Gces4VCcU0JI/view?usp=sharing)\n- `SlimSAM-77`: [SlimSAM-77 model.](https://drive.google.com/file/d/14BhU66umvY0E1FWoGsCMpLqXMNw9c3Nx/view?usp=sharing)\n\n\nAbove models can be instantiated by running\n\n``` python\nimport torch\nSlimSAM_model = torch.load(\u003cmodel_path\u003e)\nSlimSAM_model.image_encoder = SlimSAM_model.image_encoder.module\n\ndef forward(self, x):\n\n    x = self.patch_embed(x)\n    if self.pos_embed is not None:\n        x = x + self.pos_embed\n\n    for blk in self.blocks:\n        x,qkv_emb,mid_emb,x_emb = blk(x)\n\n    x = self.neck(x.permute(0, 3, 1, 2))\n    \n    return x\n\nimport types\nfuncType = types.MethodType\nSlimSAM_model.image_encoder.forward = funcType(forward, SlimSAM_model.image_encoder)\nSlimSAM_model.to(device)\nSlimSAM_model.eval()\n```\n\n\n#### Local Pruning Models:\n\n- `SlimSAM-50-uniform`: [SlimSAM-50 model.](https://drive.google.com/file/d/1Ld7Q2LY8H2nu4zB6VxwwA5npS5A9OHFq/view?usp=sharing)\n- `SlimSAM-77-uniform`: [SlimSAM-77 model.](https://drive.google.com/file/d/1OeWpfk5WhdlMz5VvYmb9gaE6suzHB0sp/view?usp=sharing)\n\n\nAbove models can be instantiated by running\n\n``` python\nimport torch\nfrom segment_anything import sam_model_registry\n\nmodel_type = 'vit_p50'\ncheckpoint = 'checkpoints/SlimSAM-50-uniform.pth'\nSlimSAM_model = sam_model_registry[model_type](checkpoint=checkpoint)\nSlimSAM_model.to(device)\nSlimSAM_model.eval()\n```\n\n\n## \u003ca name=\"Inference\"\u003e\u003c/a\u003eInference\n\nFirst download [SlimSAM-50 model](https://drive.google.com/file/d/1iCN9IW0Su0Ud_fOFoQUnTdkC3bFveMND/view?usp=sharing) or  [SlimSAM-77 model](https://drive.google.com/file/d/1L7LB6gHDzR-3D63pH9acD9E0Ul9_wMF-/view) for inference\n\n\nWe provide detailed instructions in 'inference.py' on how to use a range of prompts, including 'point' and 'box' and 'everything', for inference purposes.\n\n```\nCUDA_VISIBLE_DEVICES=0 python inference.py\n```\n\n## \u003ca name=\"Train\"\u003e\u003c/a\u003eTrain\n\nFirst download a [SAM-B model](https://drive.google.com/file/d/1CtcyOm4h9bXgBF8DEVWn3N7g9-3r4Xzz/view?usp=sharing) into 'checkpoints/' as the base model. \n\n### Step1: Embedding Pruning + Bottleneck Aligning ###\nThe model after step1 is saved as 'checkpoints/vit_b_slim_step1_.pth'\n\n```\nCUDA_VISIBLE_DEVICES=0 python prune_distill_step1.py  --traindata_path \u003ctrain_data_root\u003e --valdata_path \u003cval_data_root\u003e --prune_ratio \u003cpruning ratio\u003e --epochs \u003ctraining epochs\u003e\n```\n\n### Step2: Bottleneck Pruning + Embedding Aligning ###\nThe model after step2 is saved as 'checkpoints/vit_b_slim_step2_.pth'\n\n```\nCUDA_VISIBLE_DEVICES=0 python prune_distill_step2.py  --traindata_path \u003ctrain_data_root\u003e --valdata_path \u003cval_data_root\u003e --prune_ratio \u003cpruning ratio\u003e --epochs \u003ctraining epochs\u003e --model_path 'checkpoints/vit_b_slim_step1_.pth' \n\n```\n\nYou can adjust the training settings to meet your specific requirements. While our method demonstrates impressive performance with just 10,000 training data, incorporating additional training data will further enhance the model's effectiveness\n\n## BibTex of our SlimSAM\nIf you use SlimSAM in your research, please use the following BibTeX entry. Thank you!\n\n```bibtex\n@article{chen20230,\n  title={0.1\\% Data Makes Segment Anything Slim},\n  author={Chen, Zigeng and Fang, Gongfan and Ma, Xinyin and Wang, Xinchao},\n  journal={arXiv preprint arXiv:2312.05284},\n  year={2023}\n}\n```\n\n## Acknowledgement\n\n\u003cdetails\u003e\n\u003csummary\u003e\n\u003ca href=\"https://github.com/facebookresearch/segment-anything\"\u003eSAM\u003c/a\u003e (Segment Anything) [\u003cb\u003ebib\u003c/b\u003e]\n\u003c/summary\u003e\n\n```bibtex\n@article{kirillov2023segany,\n  title={Segment Anything}, \n  author={Kirillov, Alexander and Mintun, Eric and Ravi, Nikhila and Mao, Hanzi and Rolland, Chloe and Gustafson, Laura and Xiao, Tete and Whitehead, Spencer and Berg, Alexander C. and Lo, Wan-Yen and Doll{\\'a}r, Piotr and Girshick, Ross},\n  journal={arXiv:2304.02643},\n  year={2023}\n}\n```\n\u003c/details\u003e\n\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\n\u003ca href=\"https://github.com/VainF/Torch-Pruning\"\u003eTorch Pruning\u003c/a\u003e (DepGraph: Towards Any Structural Pruning) [\u003cb\u003ebib\u003c/b\u003e]\n\u003c/summary\u003e\n\n```bibtex\n@inproceedings{fang2023depgraph,\n  title={Depgraph: Towards any structural pruning},\n  author={Fang, Gongfan and Ma, Xinyin and Song, Mingli and Mi, Michael Bi and Wang, Xinchao},\n  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\n  pages={16091--16101},\n  year={2023}\n}\n```\n\u003c/details\u003e\n\n\n***********************************************************\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fczg1225%2FSlimSAM","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fczg1225%2FSlimSAM","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fczg1225%2FSlimSAM/lists"}