{"id":45171175,"url":"https://github.com/halleewong/ScribblePrompt","last_synced_at":"2026-03-05T06:00:40.965Z","repository":{"id":212342687,"uuid":"725677743","full_name":"halleewong/ScribblePrompt","owner":"halleewong","description":"[ECCV 2024] ScribblePrompt: Fast and Flexible Interactive Segmentation for Any Medical Image","archived":false,"fork":false,"pushed_at":"2025-07-13T23:04:23.000Z","size":28373,"stargazers_count":204,"open_issues_count":10,"forks_count":19,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-12-17T20:01:10.860Z","etag":null,"topics":["interactive-segmentation","medical-imaging","segment-anything","segmentation"],"latest_commit_sha":null,"homepage":"http://scribbleprompt.csail.mit.edu/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/halleewong.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.bib","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-11-30T16:37:47.000Z","updated_at":"2025-11-25T08:15:18.000Z","dependencies_parsed_at":"2024-01-20T00:26:36.079Z","dependency_job_id":"a1255913-9af8-41b0-a5c3-e7548ef682ad","html_url":"https://github.com/halleewong/ScribblePrompt","commit_stats":null,"previous_names":["halleewong/scribbleprompt"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/halleewong/ScribblePrompt","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/halleewong%2FScribblePrompt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/halleewong%2FScribblePrompt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/halleewong%2FScribblePrompt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/halleewong%2FScribblePrompt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/halleewong","download_url":"https://codeload.github.com/halleewong/ScribblePrompt/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/halleewong%2FScribblePrompt/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30111779,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-05T03:40:26.266Z","status":"ssl_error","status_checked_at":"2026-03-05T03:39:15.902Z","response_time":93,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["interactive-segmentation","medical-imaging","segment-anything","segmentation"],"created_at":"2026-02-20T08:05:55.701Z","updated_at":"2026-03-05T06:00:40.849Z","avatar_url":"https://github.com/halleewong.png","language":"Jupyter Notebook","funding_links":[],"categories":["public repositories"],"sub_categories":["annotation tools"],"readme":"\u003ca href=https://arxiv.org/abs/2312.07381\u003e\u003cimg src=\"https://img.shields.io/badge/arxiv-2312.07381-orange?logo=arxiv\u0026logoColor=white\"/\u003e\u003c/a\u003e\n\u003ca href=\"https://huggingface.co/spaces/halleewong/ScribblePrompt\"\u003e\u003cimg alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\"\u003e\u003c/a\u003e\n\u003ca href=\"https://colab.research.google.com/drive/14ExpVy3PjCCp4VzgTo27Yh_aLBafK8cX?usp=sharing\"\u003e\u003cimg alt=\"Colab\" src=\"https://colab.research.google.com/assets/colab-badge.svg\"\u003e\u003c/a\u003e\n\n# ScribblePrompt\n\n### [Project Page](https://scribbleprompt.csail.mit.edu) | [Paper](https://arxiv.org/abs/2312.07381) | [Online Demo](https://huggingface.co/spaces/halleewong/ScribblePrompt) | [Video](https://youtu.be/L8CiAoHzPUE)\n\nOfficial implementation of [ScribblePrompt: Fast and Flexible Interactive Segmentation for any Biomedical Image](https://arxiv.org/abs/2312.07381) accepted at ECCV 2024\n\n[Hallee E. Wong](https://halleewong.github.io/), [Marianne Rakic](https://mariannerakic.github.io/), [John Guttag](https://people.csail.mit.edu/guttag/), [Adrian V. Dalca](http://www.mit.edu/~adalca/)\n\n## Updates\n\n* (2024-12-31) Released example training code\n* (2024-12-12) Released full prompt simulation code\n* (2024-07-01) ScribblePrompt has been accepted to ECCV 2024!\n* (2024-06-17) ScribblePrompt won the [Bench-to-Bedside Award](https://dca-in-mi.github.io/awards.html) at the CVPR 2024 DCAMI Workshop!\n* (2024-04-16) Released [MedScribble](https://github.com/halleewong/ScribblePrompt/tree/main/MedScribble) -- a diverse dataset of segmentation tasks with scribble annotations\n* (2024-04-15) An updated version of the paper is on arXiv!\n* (2024-04-14) Added [Google Colab Tutorial](https://colab.research.google.com/drive/14ExpVy3PjCCp4VzgTo27Yh_aLBafK8cX?usp=sharing)\n* (2024-01-19) Released scribble simulation code\n* (2023-12-15) Released model code and weights \n* (2023-12-12) Paper and online demo released\n\n## Overview\n\n**ScribblePrompt** is an interactive segmentation tool that enables users to segment *unseen* structures in medical images using scribbles, clicks, *and* bounding boxes. \n\n![](https://github.com/halleewong/ScribblePrompt/blob/website/assets/gifs/total_segmentator.gif)\n![](https://github.com/halleewong/ScribblePrompt/blob/website/assets/gifs/wbc.gif)\n![](https://github.com/halleewong/ScribblePrompt/blob/website/assets/gifs/drive.gif)\n![](https://github.com/halleewong/ScribblePrompt/blob/website/assets/gifs/buid.gif)\n![](https://github.com/halleewong/ScribblePrompt/blob/website/assets/gifs/hipxray.gif)\n![](https://github.com/halleewong/ScribblePrompt/blob/website/assets/gifs/acdc.gif)\n\n## Try ScribblePrompt\n\n* Interactive [online demo](https://huggingface.co/spaces/halleewong/ScribblePrompt) on Hugging Face Spaces\n* See [Installation](https://github.com/halleewong/ScribblePrompt?tab=readme-ov-file#installation) and [Getting Started](https://github.com/halleewong/ScribblePrompt?tab=readme-ov-file#getting-started) for how to run the Gradio demo locally\n* Jupyter notebook [colab tutorial](https://colab.research.google.com/drive/14ExpVy3PjCCp4VzgTo27Yh_aLBafK8cX?usp=sharing) using pre-trained models\n* Jupyter notebook [tutorials](https://github.com/halleewong/ScribblePrompt/tree/main/notebooks) on training and the prompt generator code\n\n## Models\n\nWe provide [checkpoints](https://www.dropbox.com/scl/fo/zl12obhnsqc2mq7ulviq9/h?rlkey=suaj632fd9aqd6c2gtajz1ywc\u0026dl=0) for two versions of ScribblePrompt: \n\n* **ScribblePrompt-UNet** with an efficient fully-convolutional architecture  \n\n* **ScribblePrompt-SAM** based on the [Segment Anything Model](https://github.com/facebookresearch/segment-anything)\n\nBoth models have been trained with iterative **scribbles, click, and bounding box interactions** on a diverse collection of 65 medical imaging datasets with both real and synthetic labels. \n\n## MedScribble Dataset\n\nWe release MedScribble, a dataset of multi-annotator scribble annotations for diverse biomedical image segmentation tasks, under [`./MedScribble`](https://github.com/halleewong/ScribblePrompt/tree/main/MedScribble). See [the readme](https://github.com/halleewong/ScribblePrompt/tree/main/MedScribble/README.md) for more info and [`./MedScribble/tutorial.ipynb`](https://github.com/halleewong/ScribblePrompt/tree/main/MedScribble/tutorial.ipynb) for a preview of the data.  \n\n## Installation\n\nYou can install `scribbleprompt` in two ways:\n\n* **With pip**:\n\n```\n# For basic inference\npip install \"scribbleprompt @ git+https://github.com/halleewong/ScribblePrompt.git\"\n\n# For prompt simulation and training (additional dependencies)\npip install \"scribbleprompt[training] @ git+https://github.com/halleewong/ScribblePrompt.git\"\n```\n\n* **Manually**: cloning it and installing dependencies\n```\ngit clone https://github.com/halleewong/ScribblePrompt\npython -m pip install -r ./ScribblePrompt/requirements.txt\npython -m pip install -r ./ScribblePrompt/requirements_training.txt\nexport PYTHONPATH=\"$PYTHONPATH:$(realpath ./ScribblePrompt)\"\n```\n\nThe following optional dependencies are necessary for the local Gradio app demo:\n```\npip install gradio==3.40.1\n```\n\n## Getting Started\n\nFirst, download the model [checkpoints](https://www.dropbox.com/scl/fo/zl12obhnsqc2mq7ulviq9/h?rlkey=suaj632fd9aqd6c2gtajz1ywc\u0026dl=0) to `./checkpoints`.\n\nTo run an interactive demo locally: \n```\npython demos/app.py\n```\n\nTo instantiate ScribblePrompt-UNet and make a prediction:\n```\nfrom scribbleprompt import ScribblePromptUNet\n\nsp_unet = ScribblePromptUNet()\n\nmask = sp_unet.predict(\n    image,        # (B, 1, H, W) \n    point_coords, # (B, n, 2)\n    point_labels, # (B, n)\n    scribbles,    # (B, 2, H, W)\n    box,          # (B, n, 4)\n    mask_input,   # (B, 1, H, W)\n) # -\u003e (B, 1, H, W) \n```\n\nTo instantiate ScribblePrompt-SAM and make a prediction:\n```\nfrom scribbleprompt import ScribblePromptSAM\n\nsp_sam = ScribblePromptSAM()\n\nmask, img_features, low_res_logits = sp_sam.predict(\n    image,        # (B, 1, H, W) \n    point_coords, # (B, n, 2)\n    point_labels, # (B, n)\n    scribbles,    # (B, 2, H, W)\n    box,          # (B, n, 4)\n    mask_input,   # (B, 1, 256, 256)\n) # -\u003e (B, 1, H, W), (B, 16, 256, 256), (B, 1, 256, 256)\n\n```\nFor best results, `image` should have spatial dimensions $(H,W) = (128,128)$ and pixel values min-max normalized to the $[0,1]$ range. \n\nFor ScribblePrompt-UNet, `mask_input` should be the logits from the previous prediction. For ScribblePrompt-SAM, `mask_input` should be `low_res_logits` from the previous prediction. \n\n## Training\n\n\u003eNote: our training code requires the [pylot](https://github.com/JJGO/pylot) library. The inference code above does not.  We recommend installing via pip:\n\u003e```\n\u003epip install git+https://github.com/JJGO/pylot.git@87191921033c4391546fd88c5f963ccab7597995\n\u003e```\n\n\nThe configuration settings for training are controlled by yaml config files. We provide two example configs in [`./configs`](https://github.com/halleewong/ScribblePrompt/tree/main/configs) for fine-tuning from the pre-trained ScribblePrompt-UNet weights as well as training from scratch on an example dataset.\n\nTo fine-tune ScribblePrompt-UNet from the pre-trained weights:\n```\npython scribbleprompt/experiment/unet.py -config finetune_unet.yaml \n```\n\nTo train a model from scratch:\n```\npython scribbleprompt/experiment/unet.py -config train_unet.yaml \n```\n\nFor a more in-depth tutorial see [`./notebooks/training.ipynb`](https://github.com/halleewong/ScribblePrompt/tree/main/notebooks/training.ipynb).\n\n## To Do\n\n- [x] Release Gradio demo \n- [x] Release model code and weights\n- [x] Release jupyter notebook tutorial\n- [x] Release scribble simulation code\n- [x] Release MedScribble dataset\n- [x] Release training code\n- [ ] Release segmentation labels collected using ScribblePrompt\n\n## Acknowledgements\n\n* Our training code builds on the [`pylot`](https://github.com/JJGO/pylot) library for deep learning experiment management. We also make use of data augmentation code originally developed for [UniverSeg](https://github.com/JJGO/UniverSeg). Thanks to [@JJGO](https://github.com/JJGO) for sharing this code! \n\n* We use functions from [voxsynth](https://github.com/dalcalab/voxynth) for applying random deformations during scribble simulation \n\n* Code for ScribblePrompt-SAM builds on [Segment Anything](https://github.com/facebookresearch/segment-anything). Thanks to Meta AI for open-sourcing the model. \n\n\n## Citation\n\nIf you find our work or any of our materials useful, please cite our paper:\n```\n@article{wong2024scribbleprompt,\n  title={ScribblePrompt: Fast and Flexible Interactive Segmentation for Any Biomedical Image},\n  author={Hallee E. Wong and Marianne Rakic and John Guttag and Adrian V. Dalca},\n  journal={European Conference on Computer Vision (ECCV)},\n  year={2024},\n}\n```\n\n## License\n\nCode for this project is released under the [Apache 2.0 License](https://github.com/halleewong/ScribblePrompt/blob/main/LICENSE) \n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhalleewong%2FScribblePrompt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhalleewong%2FScribblePrompt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhalleewong%2FScribblePrompt/lists"}