{"id":22068421,"url":"https://github.com/uncbiag/SegNext","last_synced_at":"2025-07-24T06:31:12.755Z","repository":{"id":230461252,"uuid":"769763235","full_name":"uncbiag/SegNext","owner":"uncbiag","description":"Rethinking Interactive Image Segmentation with Low Latency, High Quality, and Diverse Prompts (CVPR 2024)","archived":false,"fork":false,"pushed_at":"2025-07-08T06:07:50.000Z","size":92356,"stargazers_count":91,"open_issues_count":3,"forks_count":13,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-07-08T07:01:06.388Z","etag":null,"topics":["interactive-image-segmentation","segment-anything","vision-transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/uncbiag.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-10T01:34:22.000Z","updated_at":"2025-07-08T06:07:53.000Z","dependencies_parsed_at":"2024-03-29T22:26:31.630Z","dependency_job_id":"e3df200b-b53a-4c0e-ae26-b03cd0737223","html_url":"https://github.com/uncbiag/SegNext","commit_stats":null,"previous_names":["uncbiag/segnext"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/uncbiag/SegNext","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uncbiag%2FSegNext","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uncbiag%2FSegNext/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uncbiag%2FSegNext/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uncbiag%2FSegNext/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/uncbiag","download_url":"https://codeload.github.com/uncbiag/SegNext/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uncbiag%2FSegNext/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266802637,"owners_count":23986384,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-24T02:00:09.469Z","response_time":99,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["interactive-image-segmentation","segment-anything","vision-transformers"],"created_at":"2024-11-30T20:04:01.219Z","updated_at":"2025-07-24T06:31:12.742Z","avatar_url":"https://github.com/uncbiag.png","language":"Python","funding_links":[],"categories":["Paper List"],"sub_categories":["Follow-up Papers"],"readme":"# Rethinking Interactive Image Segmentation with Low Latency, High Quality, and Diverse Prompts\nPytorch implementation for paper [Rethinking Interactive Image Segmentation with Low Latency, High Quality, and Diverse Prompts](https://arxiv.org/abs/2404.00741), CVPR 2024. \u003cbr\u003e\nQin Liu, Jaemin Cho, Mohit Bansal, Marc Niethammer \u003cbr\u003e\nUNC-Chapel Hill\n#### [Paper](https://arxiv.org/pdf/2404.00741.pdf) | [Project](https://uncbiag.github.io/SegNext/) | [Demos](https://drive.google.com/drive/folders/13tOhSYFCY2Ue8QR5rR8EEWHXGE75Zkxo?usp=sharing)\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./assets/medal.gif\" alt=\"drawing\", height=\"170\"/\u003e\n  \u003cimg src=\"./assets/bicyclestand.gif\" alt=\"drawing\", height=\"170\"/\u003e\n  \u003cimg src=\"./assets/crack.gif\" alt=\"drawing\", height=\"170\"/\u003e\n\n\u003c/p\u003e\n\n## Installation\nThe code is tested with ``python=3.10``, ``torch=2.2.0``, ``torchvision=0.17.0``.\n```\ngit clone https://github.com/uncbiag/SegNext\ncd SegNext\n```\nNow, create a new conda environment and install required packages accordingly.\n```\nconda create -n segnext python=3.10\nconda activate segnext\nconda install pytorch==2.2.0 torchvision==0.17.0 torchaudio==2.2.0 pytorch-cuda=11.8 -c pytorch -c nvidia\npip install -r requirements.txt\n```\n## Getting Started\nFirst, download three model weights: vitb_sax1 (408M), vitb_sax2 (435M), and vitb_sax2_ft (435M). These weights will be automatically saved to the ``weights`` folder.\n```\npython download.py\n``` \nRun interactive GUI with the downloaded weights. The ``assets`` contains images for demo.\n```\n./run_demo.sh\n``` \n\n## Datasets\nWe train and test our method on three datasets: DAVIS, COCO+LVIS, and HQSeg-44K.\n\n| Dataset   |                      Description             |           Download Link              |\n|-----------|----------------------------------------------|:------------------------------------:|\n|DAVIS      |  345 images with one object each (test)      |  [DAVIS.zip (43 MB)][DAVIS]          |\n|HQSeg-44K  |  44320 images (train); 1537 images (val)     |  [official site][HQSeg]              |\n|COCO+LVIS* |  99k images with 1.5M instances (train)      |  [original LVIS images][LVIS] + \u003cbr\u003e [combined annotations][COCOLVIS_annotation] |\n\n[HQSeg]: https://huggingface.co/sam-hq-team/sam-hq-training/tree/main/data\n[LVIS]: https://www.lvisdataset.org/dataset\n[DAVIS]: https://drive.google.com/uc?export=download\u0026id=1kyjN6EJSjwGnzSJxVjm3Pl2-XAjV7tac\n[COCOLVIS_annotation]: https://drive.google.com/uc?export=download\u0026id=17z9aZPlRv8vpU1AEz_M0WuZC6uBkqEWE\n\nDon't forget to change the paths to the datasets in [config.yml](config.yml) after downloading and unpacking.\n\n(*) To prepare COCO+LVIS, you need to download original LVIS v1.0, then download and unpack \npre-processed annotations that are obtained by combining COCO and LVIS dataset into the folder with LVIS v1.0. (The combined annotations are prepared by [RITM](https://github.com/SamsungLabs/ritm_interactive_segmentation).)\n\n## Evaluation\nWe provide a script (``run_eval.sh``) to evaluate our presented models. The following command runs the NoC evaluation on all test datasets.\n```\npython ./segnext/scripts/evaluate_model.py --gpus=0 --checkpoint=./weights/vitb_sa2_cocolvis_hq44k_epoch_0.pth --datasets=DAVIS,HQSeg44K\n```\n\n\u003ctable\u003e\n    \u003cthead align=\"center\"\u003e\n        \u003ctr\u003e\n            \u003cth rowspan=\"2\"\u003e\u003cspan style=\"font-weight:bold\"\u003eTrain\u003c/span\u003e\u003cbr\u003e\u003cspan style=\"font-weight:bold\"\u003eDataset\u003c/span\u003e\u003c/th\u003e\n            \u003cth rowspan=\"2\"\u003eModel\u003c/th\u003e\n            \u003cth colspan=\"4\"\u003eHQSeg-44K\u003c/th\u003e\n            \u003cth colspan=\"4\"\u003eDAVIS\u003c/th\u003e\n        \u003c/tr\u003e\n        \u003ctr\u003e\n            \u003ctd\u003e5-mIoU\u003c/td\u003e\n            \u003ctd\u003eNoC90\u003c/td\u003e\n            \u003ctd\u003eNoC95\u003c/td\u003e\n            \u003ctd\u003eNoF95\u003c/td\u003e\n            \u003ctd\u003e5-mIoU\u003c/td\u003e\n            \u003ctd\u003eNoC90\u003c/td\u003e\n            \u003ctd\u003eNoC95\u003c/td\u003e\n            \u003ctd\u003eNoF95\u003c/td\u003e\n        \u003c/tr\u003e\n    \u003c/thead\u003e\n    \u003ctbody align=\"center\"\u003e\n        \u003ctr\u003e\n            \u003ctd rowspan=\"1\"\u003eC+L\u003c/td\u003e\n            \u003ctd align=\"left\"\u003e\u003ca href=\"https://drive.google.com/uc?export=download\u0026id=1eqkd5-J9MELGIw2WRcT5hejsnGc5oO30\"\u003evitb-sax1 (408 MB)\u003c/a\u003e\u003c/td\u003e\n            \u003ctd\u003e85.41\u003c/td\u003e\n            \u003ctd\u003e7.47\u003c/td\u003e\n            \u003ctd\u003e11.94\u003c/td\u003e\n            \u003ctd\u003e731\u003c/td\u003e\n            \u003ctd\u003e90.13\u003c/td\u003e\n            \u003ctd\u003e5.46\u003c/td\u003e\n            \u003ctd\u003e13.31\u003c/td\u003e\n            \u003ctd\u003e177\u003c/td\u003e\n        \u003c/tr\u003e\n        \u003ctr\u003e\n            \u003ctd rowspan=\"1\"\u003eC+L\u003c/td\u003e\n            \u003ctd align=\"left\"\u003e\u003ca href=\"https://drive.google.com/uc?export=download\u0026id=1oxwCm4bFby6RgltO_tl54BqRN9tojylT\"\u003evitb-sax2 (435 MB)\u003c/a\u003e\u003c/td\u003e\n            \u003ctd\u003e85.71\u003c/td\u003e\n            \u003ctd\u003e7.18\u003c/td\u003e\n            \u003ctd\u003e11.52\u003c/td\u003e\n            \u003ctd\u003e700\u003c/td\u003e\n            \u003ctd\u003e89.85\u003c/td\u003e\n            \u003ctd\u003e5.34\u003c/td\u003e\n            \u003ctd\u003e12.80\u003c/td\u003e\n            \u003ctd\u003e163\u003c/td\u003e\n        \u003c/tr\u003e\n        \u003ctr\u003e\n            \u003ctd rowspan=\"1\"\u003eC+L+HQ\u003c/td\u003e\n            \u003ctd align=\"left\"\u003e\u003ca href=\"https://drive.google.com/uc?export=download\u0026id=1yDN3mwBBO33TlA0KRdO2s07Q5HWXR6nt\"\u003evitb-sax2 (435 MB)\u003c/a\u003e\u003c/td\u003e\n            \u003ctd\u003e91.75\u003c/td\u003e\n            \u003ctd\u003e5.32\u003c/td\u003e\n            \u003ctd\u003e9.42\u003c/td\u003e\n            \u003ctd\u003e583\u003c/td\u003e\n            \u003ctd\u003e91.87\u003c/td\u003e\n            \u003ctd\u003e4.43\u003c/td\u003e\n            \u003ctd\u003e10.73\u003c/td\u003e\n            \u003ctd\u003e123\u003c/td\u003e\n        \u003c/tr\u003e\n    \u003c/tbody\u003e\n\u003c/table\u003e\n\nFor SAT latency evaluation, please refer to [eval_sat_latency.ipynb](./notebooks/eval_sat_latency.ipynb).\n\n## Training\nWe provide a script (``run_train.sh``) for training our models on the HQSeg-44K dataset. You can start training with the following commands. By default we use 4 A6000 GPUs for training.\n```\n# train vitb-sax1 model on coco+lvis \nMODEL_CONFIG=./segnext/models/default/plainvit_base1024_cocolvis_sax1.py\ntorchrun --nproc-per-node=4 --master-port 29504 ./segnext/train.py ${MODEL_CONFIG} --batch-size=16 --gpus=0,1,2,3\n\n# train vitb-sax2 model on coco+lvis \nMODEL_CONFIG=./segnext/models/default/plainvit_base1024_cocolvis_sax2.py\ntorchrun --nproc-per-node=4 --master-port 29505 ./segnext/train.py ${MODEL_CONFIG} --batch-size=16 --gpus=0,1,2,3\n\n# finetune vitb-sax2 model on hqseg-44k \nMODEL_CONFIG=./segnext/models/default/plainvit_base1024_hqseg44k_sax2.py\ntorchrun --nproc-per-node=4 --master-port 29506 ./segnext/train.py ${MODEL_CONFIG} --batch-size=12 --gpus=0,1,2,3 --weights ./weights/vitb_sa2_cocolvis_epoch_90.pth\n\n```\n\n## Citation\n```bibtex\n@article{liu2024rethinking,\n  title={Rethinking Interactive Image Segmentation with Low Latency, High Quality, and Diverse Prompts},\n  author={Liu, Qin and Cho, Jaemin and Bansal, Mohit and Niethammer, Marc},\n  journal={arXiv preprint arXiv:2404.00741},\n  year={2024}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Funcbiag%2FSegNext","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Funcbiag%2FSegNext","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Funcbiag%2FSegNext/lists"}