{"id":19327318,"url":"https://github.com/hkchengrex/scribble-to-mask","last_synced_at":"2025-04-22T21:30:23.570Z","repository":{"id":59958328,"uuid":"345923650","full_name":"hkchengrex/Scribble-to-Mask","owner":"hkchengrex","description":"[CVPR 2021] MiVOS - Scribble to Mask module","archived":false,"fork":false,"pushed_at":"2024-02-16T05:42:07.000Z","size":174,"stargazers_count":87,"open_issues_count":2,"forks_count":15,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-02T04:03:25.777Z","etag":null,"topics":["computer-vision","cvpr2021","deep-learning","interactive-segmentation","pytorch","segmentation"],"latest_commit_sha":null,"homepage":"https://hkchengrex.github.io/MiVOS/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hkchengrex.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-03-09T07:45:27.000Z","updated_at":"2025-03-30T04:23:48.000Z","dependencies_parsed_at":"2023-01-23T03:00:41.319Z","dependency_job_id":null,"html_url":"https://github.com/hkchengrex/Scribble-to-Mask","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hkchengrex%2FScribble-to-Mask","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hkchengrex%2FScribble-to-Mask/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hkchengrex%2FScribble-to-Mask/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hkchengrex%2FScribble-to-Mask/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hkchengrex","download_url":"https://codeload.github.com/hkchengrex/Scribble-to-Mask/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250328046,"owners_count":21412546,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","cvpr2021","deep-learning","interactive-segmentation","pytorch","segmentation"],"created_at":"2024-11-10T02:16:46.362Z","updated_at":"2025-04-22T21:30:23.303Z","avatar_url":"https://github.com/hkchengrex.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MiVOS (CVPR 2021) - Scribble To Mask\n\n[Ho Kei Cheng](https://hkchengrex.github.io/), Yu-Wing Tai, Chi-Keung Tang\n\n[[arXiv]](https://arxiv.org/abs/2103.07941) [[Paper PDF]](https://arxiv.org/pdf/2103.07941.pdf) [[Project Page]](https://hkchengrex.github.io/MiVOS/)\n\nA simplistic network that turns scribbles to mask. It supports multi-object segmentation using soft-aggregation. Don't expect SOTA results from this model!\n\n![Ex1](https://imgur.com/HesuB4x.gif) ![Ex2](https://imgur.com/NmCrCE1.gif)\n\n## Overall structure and capabilities\n\n| | [MiVOS](https://github.com/hkchengrex/MiVOS) | [Mask-Propagation](https://github.com/hkchengrex/Mask-Propagation)| [Scribble-to-Mask](https://github.com/hkchengrex/Scribble-to-Mask)  |\n| ------------- |:-------------:|:-----:|:-----:|\n| DAVIS/YouTube semi-supervised evaluation | :x: | :heavy_check_mark: | :x: |\n| DAVIS interactive evaluation | :heavy_check_mark: | :x: | :x: |\n| User interaction GUI tool | :heavy_check_mark: | :x: | :x: |\n| Dense Correspondences | :x: | :heavy_check_mark: | :x: |\n| Train propagation module | :x: | :heavy_check_mark: | :x: |\n| Train S2M (interaction) module | :x: | :x: | :heavy_check_mark: |\n| Train fusion module | :heavy_check_mark: | :x: | :x: |\n| Generate more synthetic data | :heavy_check_mark: | :x: | :x: |\n\n## Requirements\n\nThe package versions shown here are the ones that I used. You might not need the exact versions.\n\n- PyTorch `1.6.0`\n- torchvision `0.7.0`\n- opencv-contrib `4.2.0`\n- davis-interactive (\u003chttps://github.com/albertomontesg/davis-interactive\u003e)\n- gitpython for training\n- gdown for downloading pretrained models\n\nRefer to the official [PyTorch guide](\u003chttps://pytorch.org/\u003e) for installing PyTorch/torchvision. The rest can be installed by:\n\n`pip install opencv-contrib-python gitpython gdown`\n\n## Pretrained model\n\n[Download](https://drive.google.com/file/d/1HKwklVey3P2jmmdmrACFlkXtcvNxbKMM/view?usp=sharing) and put the model in `./saves/`. Alternatively use the provided `download_model.py`.\n\n[[OneDrive Mirror]](https://hkustconnect-my.sharepoint.com/:f:/g/personal/hkchengad_connect_ust_hk/EjHifAlvYUFPlEG2qBr-GGQBb1XyzxUvizJiQKBf8te2Cw?e=a6mxKz)\n\n## Interactive GUI\n\n`python interactive.py --image \u003cimage\u003e`\n\nControls:\n\n```bash\nMouse Left - Draw scribbles\nMouse middle key - Switch positive/negative\nKey f - Commit changes, clear scribbles\nKey r - Clear everything\nKey d - Switch between overlay/mask view\nKey s - Save masks into a temporary output folder (./output/)\n```\n\n## Known issues\n\nThe model almost always needs to focus on at least one object. It is very difficult to erase all existing masks from an image using scribbles.\n\n## Training\n\n### Datasets\n\n1. Download and extract [LVIS](https://www.lvisdataset.org/dataset) training set.\n2. Download and extract [a set of static image segmentation datasets](https://drive.google.com/file/d/1wUJq3HcLdN-z1t4CsUhjeZ9BVDb9YKLd/view?usp=sharing). These are already downloaded for you if you used the `download_datasets.py` in [Mask-Propagation](https://github.com/hkchengrex/Mask-Propagation).\n\n```bash\n├── lvis\n│   ├── lvis_v1_train.json\n│   └── train2017\n├── Scribble-to-Mask\n└── static\n    ├── BIG_small\n    └── ...\n```\n\n### Commands\n\nUse the `deeplabv3plus_resnet50` pretrained model provided [here](https://github.com/VainF/DeepLabV3Plus-Pytorch).\n\n`CUDA_VISIBLE_DEVICES=0,1 OMP_NUM_THREADS=4 python -m torch.distributed.launch --master_port 9842 --nproc_per_node=2 train.py --id s2m --load_deeplab \u003cpath_to_deeplab.pth\u003e`\n\n## Credit\n\nDeeplab implementation and pretrained model: \u003chttps://github.com/VainF/DeepLabV3Plus-Pytorch\u003e.\n\n## Citation\n\nPlease cite our paper if you find this repo useful!\n\n```bibtex\n@inproceedings{cheng2021mivos,\n  title={Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusion},\n  author={Cheng, Ho Kei and Tai, Yu-Wing and Tang, Chi-Keung},\n  booktitle={CVPR},\n  year={2021}\n}\n```\n\nContact: \u003chkchengrex@gmail.com\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhkchengrex%2Fscribble-to-mask","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhkchengrex%2Fscribble-to-mask","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhkchengrex%2Fscribble-to-mask/lists"}