{"id":49501209,"url":"https://github.com/heyoeyo/muggled_sam","last_synced_at":"2026-05-01T12:34:32.407Z","repository":{"id":250061756,"uuid":"833358036","full_name":"heyoeyo/muggled_sam","owner":"heyoeyo","description":"Muggled SAM: Segmentation without the magic","archived":false,"fork":false,"pushed_at":"2026-04-30T20:14:21.000Z","size":3522,"stargazers_count":223,"open_issues_count":1,"forks_count":30,"subscribers_count":8,"default_branch":"main","last_synced_at":"2026-04-30T22:13:12.049Z","etag":null,"topics":["deeplearning","educational-project","object-detection","segment-anything","segmentation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/heyoeyo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-07-24T22:10:36.000Z","updated_at":"2026-04-30T20:15:03.000Z","dependencies_parsed_at":"2024-08-05T00:20:47.957Z","dependency_job_id":"1e77a37d-47c6-4a6a-9c9d-f4d26a32ff6a","html_url":"https://github.com/heyoeyo/muggled_sam","commit_stats":null,"previous_names":["heyoeyo/muggled_sam"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/heyoeyo/muggled_sam","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/heyoeyo%2Fmuggled_sam","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/heyoeyo%2Fmuggled_sam/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/heyoeyo%2Fmuggled_sam/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/heyoeyo%2Fmuggled_sam/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/heyoeyo","download_url":"https://codeload.github.com/heyoeyo/muggled_sam/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/heyoeyo%2Fmuggled_sam/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32497814,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-30T13:12:12.517Z","status":"online","status_checked_at":"2026-05-01T02:00:05.856Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deeplearning","educational-project","object-detection","segment-anything","segmentation"],"created_at":"2026-05-01T12:34:31.678Z","updated_at":"2026-05-01T12:34:32.389Z","avatar_url":"https://github.com/heyoeyo.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MuggledSAM\n\nThis repo contains a simplified implementation of the awesome 'Segment Anything' models from [facebookresearch](https://ai.meta.com/research/#projects) ([SAM1](https://github.com/facebookresearch/segment-anything), [SAM2](https://github.com/facebookresearch/sam2) \u0026 [SAM3](https://github.com/facebookresearch/sam3)), with the intention of [removing the magic](https://en.wikipedia.org/wiki/Muggle) from the original code base to make it easier to understand.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\".readme_assets/demo_anim.gif\"\u003e\n\u003c/p\u003e\n\nAs a result of simplifying the code, this repo supports video tracking on _arbitrarily long_ videos (using the v2 or v3 models). It also supports adjustments to the input image resolution, which can [speed up](https://github.com/heyoeyo/muggled_sam/tree/main/simple_examples#video-segmentation) model inference in some cases.\n\nThere is a written walkthrough explaining the structure of the [SAMv1 model](https://github.com/heyoeyo/muggled_sam/tree/main/muggled_sam/v1_sam), with documentation for v2/v3 on the way!\n\n\n## Getting started\n\nThis repo includes three demo scripts, [run_image.py](https://github.com/heyoeyo/muggled_sam/blob/main/run_image.py), [run_video.py](https://github.com/heyoeyo/muggled_sam/blob/main/run_video.py) and [run_detections.py](https://github.com/heyoeyo/muggled_sam/blob/main/run_detections.py), along with a number of [simple examples](https://github.com/heyoeyo/muggled_sam/tree/main/simple_examples) and [experiments](https://github.com/heyoeyo/muggled_sam/tree/main/experiments). There are also some [training](https://github.com/heyoeyo/muggled_sam/tree/main/training) scripts available, specifically for SAMv3. To use these scripts, you'll first need to have [Python](https://www.python.org/) (v3.10+) installed, then set up a virtual environment and install some additional requirements.\n\n### Install\nUsing a terminal, first create and activate a virtual environment (do this inside the repo folder after [cloning/downloading](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository) it):\n```bash\n# For linux or mac:\npython3 -m venv .env\nsource .env/bin/activate\n\n# For windows (cmd):\npython -m venv .env\n.env\\Scripts\\activate.bat\n```\n\nThen install the requirements (or you could install them manually from the [requirements.txt](https://github.com/heyoeyo/muggled_sam/blob/main/requirements.txt) file):\n```bash\npip install -r requirements.txt\n```\n\n\u003cdetails\u003e\n\u003csummary\u003eAdditional info for GPU usage\u003c/summary\u003e\n\nIf you're using Windows and want to use an Nvidia GPU or if you're on Linux and don't have a GPU, you'll need to use a slightly different install command to make use of your hardware setup. You can use the [Pytorch installer guide](https://pytorch.org/get-started/locally/) to figure out the command to use. For example, for GPU use on Windows it may look something like:\n```bash\npip3 uninstall torch  # \u003c-- Do this first if you already installed from the requirements.txt file\npip3 install torch --index-url https://download.pytorch.org/whl/cu121\n```\n\n**Note**: With the Windows install as-is, you may get an error about a `missing c10.dll` dependency. Downloading and installing this [mysterious .exe file](https://aka.ms/vs/16/release/vc_redist.x64.exe) seems to fix the problem.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e Alternative installations\u003c/summary\u003e\n\n#### Install from Github\n\nIt's also possible to install this repo directly from Github:\n```bash\n# Be sure to activate a virtual environment before installing!\npip install git+https://github.com/heyoeyo/muggled_sam\n```\nThis will make the repo available as a library (e.g. to use models in another project), though the demo scripts will not be available through this installation method.\n\n\n#### Other package managers\n\nUsing package managers like [conda](https://docs.conda.io/projects/conda/en/stable/user-guide/getting-started.html) or [uv](https://docs.astral.sh/uv/) can also work but may require slightly different installation commands. If you have an existing python environment set up for another pytorch image model, it can probably be reused with this repo.\n\n\u003c/details\u003e\n\n### Model Weights\n\nBefore you can run a model, you'll need to download it's weights. There are 3 supported SAMv1 models (vit-base, vit-large and vit-huge), four v2/v2.1 models (tiny, small, base-plus and large) and 1 SAMv3/v3.1 model. This repo uses the exact same weights as the original implementations (or any fine-tuned variant of the original models), which can be downloaded from the **Model Description** section of the [SAMv2 repo](https://github.com/facebookresearch/sam2?tab=readme-ov-file#model-description) (config files are not needed, only the checkpoints) and the **Model Checkpoints** section of the [SAMv1 repo](https://github.com/facebookresearch/segment-anything?tab=readme-ov-file#model-checkpoints). The v3 weights require (as of Dec 2025) signing an agreement before downloading, see the [SAMv3](https://github.com/facebookresearch/sam3?tab=readme-ov-file#getting-started) or [SAMv3.1](https://github.com/facebookresearch/sam3/blob/main/RELEASE_SAM3p1.md#new-checkpoints) repo for more details.\n\nAfter downloading a model file, you can place it in the `model_weights` folder of this repo or otherwise just keep note of the file path, since you'll need to provide this when running the demo scripts. If you do place the file in the [model_weights](https://github.com/heyoeyo/muggled_sam/tree/main/model_weights) folder, then it will auto-load when running the scripts.\n\n\u003cdetails\u003e\n\n\u003csummary\u003eDirect download links\u003c/summary\u003e\n\nThe tables below include direct download links to all of the supported models. **Note:** These are all links to the original repos, none of these files belong to MuggledSAM!\n\n| SAMv3 Models | Size (MB) |\n| -----| -----|\n| [sam3.1_multiplex](https://huggingface.co/facebook/sam3.1/resolve/main/sam3.1_multiplex.pt?download=true) | 3340 |\n| [sam3](https://huggingface.co/facebook/sam3/resolve/main/sam3.pt?download=true) | 3290 |\n\n| SAMv2 Models | Size (MB) |\n| -----| -----|\n| [sam2.1_hiera_tiny](https://dl.fbaipublicfiles.com/segment_anything_2/092824/sam2.1_hiera_tiny.pt) | 160 |\n| [sam2_hiera_tiny](https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_tiny.pt) | 160 |\n| [sam2.1_hiera_small](https://dl.fbaipublicfiles.com/segment_anything_2/092824/sam2.1_hiera_small.pt) | 185 |\n| [sam2_hiera_small](https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_small.pt) | 185 |\n| [sam2.1_hiera_base_plus](https://dl.fbaipublicfiles.com/segment_anything_2/092824/sam2.1_hiera_base_plus.pt) | 325 |\n| [sam2_hiera_base_plus](https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_base_plus.pt) | 325 |\n| [sam2.1_hiera_large](https://dl.fbaipublicfiles.com/segment_anything_2/092824/sam2.1_hiera_large.pt) | 900 |\n| [sam2_hiera_large](https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_large.pt) | 900 |\n\n| SAMv1 Models | Size (MB) |\n| -----| -----|\n| [sam-vit-base](https://dl.fbaipublicfiles.com/segment_anything/sam_vit_b_01ec64.pth) | 375 |\n| [sam-vit-large](https://dl.fbaipublicfiles.com/segment_anything/sam_vit_l_0b3195.pth) | 1250 |\n| [sam-vit-huge](https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth) | 2560 |\n\n\u003c/details\u003e\n\n### Basic Example\nHere's an [example](https://github.com/heyoeyo/muggled_sam/tree/main/simple_examples/image_segmentation.py) of using the model to generate masks from an image:\n```python\nimport cv2\nfrom muggled_sam.make_sam import make_sam_from_state_dict\n\n# Define prompts using 0-to-1 xy coordinates\n# -\u003e (0,0) is top-left, (1,1) is bottom-right\nbox_xy1xy2s = []  # Example [((0.25, 0.25), (0.75, 0.75))]\nfg_xys = [(0.5, 0.5)]\nbg_xys = []\n\n# Load image \u0026 model\nimage_bgr = cv2.imread(\"/path/to/image.jpg\")\n_, model = make_sam_from_state_dict(\"/path/to/model.pth\")\n\n# Process data\nencoded_img, _, _ = model.encode_image(image_bgr)\nencoded_prompts = model.encode_prompts(box_xy1xy2s, fg_xys, bg_xys)\nmask_preds, iou_preds = model.generate_masks(encoded_img, encoded_prompts)\n```\n\nMore examples can be found in the [simple_examples](https://github.com/heyoeyo/muggled_sam/tree/main/simple_examples) folder.\n\n## Run Image\n\n_(Supports SAMv1, SAMv2, SAMv3)_\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\".readme_assets/run_image_anim.gif\"\u003e\n\u003c/p\u003e\n\nThe `run_image.py` script will run the segment-anything model on a single image with an interactive UI running locally. To use the script, make sure you've activated the virtual environment (from the installation step) and then, from the repo folder use:\n```bash\npython run_image.py\n```\n\nYou can also add  `--help` to the end of this command to see a list of additional flags you can set when running this script. For example, two interesting options are the `--crop` flag to interactively crop an image prior to processing and the `-b` flag, which can change the processing resolution of the model.\n\nIf you don't provide an image path (using the `-i` flag), then you will be asked to provide one when you run the script, likewise for a path to the model weights. Afterwards, a window will pop-up, with options for how to 'prompt' the model (e.g. bounding boxes or clicking to add points) along the top and various sliders to alter the segmentation results at the bottom. Results can be saved by pressing the `s` key.\n\nCheck out the [image segmentation](https://github.com/heyoeyo/muggled_sam/tree/main/simple_examples/image_segmentation.py) example for a simpler, hackable version of this functionality.\n\n## Run Video (or webcam)\n\n_(Supports SAMv2, SAMv3)_\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\".readme_assets/run_video_anim.gif\"\u003e\n\u003c/p\u003e\n\n\nThe `run_video.py` script allows for segmentation of videos using an interactive UI running locally. To use the script, make sure you've activated the virtual environment (from the installation step) and then, from the repo folder use:\n```bash\npython run_video.py\n```\n\nAs with the image script, you can add `--help` to the end of this command to see a list of additional flags. For example, you can add the flag `--use_webcam` to run segmentation on a live webcam feed. Using `-b 512` to reduce the processing resolution can provide a significant speed up if needed (box prompting works better at reduced resolutions btw!). \n\nThe segmentation results can be saved as .pngs (in a [tarfile](https://en.wikipedia.org/wiki/Tar_(computing))) or as an .mp4 video file (using `--ffmpeg`), please see the [video from frames](.readme_assets/video_from_frames.md) explainer for more info.\n\nThis script is a messy work-in-progress for now, more features \u0026 stability updates to come! If you'd like a more hackable solution, check out the (much easier to follow) [video segmentation example](https://github.com/heyoeyo/muggled_sam/blob/main/simple_examples/video_segmentation.py).\n\n## Run Detections\n\n_(Supports SAMv3)_\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\".readme_assets/run_detections_anim.gif\"\u003e\n\u003c/p\u003e\n\nThe `run_detections.py` script provides an interactive visualization of the output from the SAMv3 detection model. This model is capable of detecting many objects in an image using a text prompt or by using points/bounding-boxes around a reference object in the image. As usual, make sure you've activated the virtual environment from installation and then (in a terminal) use:\n```bash\npython run_detections.py\n```\nAs with the other scripts, you can add `--help` to the end of this command to see additional flags. One interesting flag is `-r /path/to/other_image.jpg` which enables the use of a separate 'reference' image, so that points/bounding boxes around objects in one image can be used to segment objects in another image (though the model doesn't officially support this!). There is also (experimental) support for compilation using the `--compile` flag.\n\nThe basic usage of this script is to input point or box prompts on the left image while the corresponding detections will be displayed on the right. When using text prompts, the UI will 'freeze' and hand off input to your terminal (the one that launched the script) for text input. Entering a blank prompt will return control to the UI.\n\nWhile this script only works with images, the detection capabilities of SAMv3 can be used with video tracking. See the [video segmentation from detections](https://github.com/heyoeyo/muggled_sam/tree/main/simple_examples#video-segmentation-from-detections) example script for more info.\n\nFor a more hackable version of cross-image detection (i.e. using an object from one image to detect objects in other images), see the [cross-image object detection](https://github.com/heyoeyo/muggled_sam/tree/main/simple_examples#object-detection-cross-image) example.\n\n\n# Changelog\n\nOver time there have been several breaking changes to the SAM implementations in MuggledSAM, these are documented in the [changelog](https://github.com/heyoeyo/muggled_sam/blob/main/.readme_assets/CHANGELOG.md).\n\n\n# Acknowledgements\n\nThe code in this repo is entirely based off the original segment-anything github repos:\n\n[facebookresearch/segment-anything](https://github.com/facebookresearch/segment-anything)\n```\n@article{kirillov2023segany,\n  title={Segment Anything},\n  author={Kirillov, Alexander and Mintun, Eric and Ravi, Nikhila and Mao, Hanzi and Rolland, Chloe and Gustafson, Laura and Xiao, Tete and Whitehead, Spencer and Berg, Alexander C. and Lo, Wan-Yen and Doll{\\'a}r, Piotr and Girshick, Ross},\n  journal={arXiv:2304.02643},\n  year={2023}\n}\n```\n\n\n[facebookresearch/sam2](https://github.com/facebookresearch/sam2)\n```bibtex\n@article{ravi2024sam2,\n  title={SAM 2: Segment Anything in Images and Videos},\n  author={Ravi, Nikhila and Gabeur, Valentin and Hu, Yuan-Ting and Hu, Ronghang and Ryali, Chaitanya and Ma, Tengyu and Khedr, Haitham and R{\\\"a}dle, Roman and Rolland, Chloe and Gustafson, Laura and Mintun, Eric and Pan, Junting and Alwala, Kalyan Vasudev and Carion, Nicolas and Wu, Chao-Yuan and Girshick, Ross and Doll{\\'a}r, Piotr and Feichtenhofer, Christoph},\n  journal={arXiv preprint},\n  year={2024}\n}\n```\n\n[facebookresearch/sam3](https://github.com/facebookresearch/sam3?tab=readme-ov-file#getting-started)\n```bibtex\n@misc{\n  carion2025sam3segmentconcepts,\n  title={SAM 3: Segment Anything with Concepts},\n  author={Nicolas Carion and Laura Gustafson and Yuan-Ting Hu and Shoubhik Debnath and Ronghang Hu and Didac Suris and Chaitanya Ryali and Kalyan Vasudev Alwala and Haitham Khedr and Andrew Huang and Jie Lei and Tengyu Ma and Baishan Guo and Arpit Kalla and Markus Marks and Joseph Greer and Meng Wang and Peize Sun and Roman Rädle and Triantafyllos Afouras and Effrosyni Mavroudi and Katherine Xu and Tsung-Han Wu and Yu Zhou and Liliane Momeni and Rishi Hazra and Shuangrui Ding and Sagar Vaze and Francois Porcher and Feng Li and Siyuan Li and Aishwarya Kamath and Ho Kei Cheng and Piotr Dollár and Nikhila Ravi and Kate Saenko and Pengchuan Zhang and Christoph Feichtenhofer},\n  year={2025},\n  eprint={2511.16719},\n  archivePrefix={arXiv},\n  primaryClass={cs.CV},\n  url={https://arxiv.org/abs/2511.16719},\n}\n```\n\n\n# TODOs\n- Add model structure documentation\n- Inevitable bugfixes","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fheyoeyo%2Fmuggled_sam","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fheyoeyo%2Fmuggled_sam","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fheyoeyo%2Fmuggled_sam/lists"}