{"id":19181719,"url":"https://github.com/kdexd/coco-rem","last_synced_at":"2025-05-07T23:07:12.433Z","repository":{"id":230140610,"uuid":"778592619","full_name":"kdexd/coco-rem","owner":"kdexd","description":"Code for the paper \"Benchmarking Object Detectors with COCO: A New Path Forward.\"","archived":false,"fork":false,"pushed_at":"2024-07-13T21:47:59.000Z","size":833,"stargazers_count":27,"open_issues_count":2,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-05-07T23:07:07.830Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kdexd.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-03-28T02:27:52.000Z","updated_at":"2025-04-19T01:23:52.000Z","dependencies_parsed_at":null,"dependency_job_id":"2a1fa3c6-195a-4b97-add9-4f78a73d23ea","html_url":"https://github.com/kdexd/coco-rem","commit_stats":null,"previous_names":["kdexd/coco-rem"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kdexd%2Fcoco-rem","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kdexd%2Fcoco-rem/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kdexd%2Fcoco-rem/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kdexd%2Fcoco-rem/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kdexd","download_url":"https://codeload.github.com/kdexd/coco-rem/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252968118,"owners_count":21833251,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T10:54:21.300Z","updated_at":"2025-05-07T23:07:12.401Z","avatar_url":"https://github.com/kdexd.png","language":"Python","funding_links":[],"categories":["Paper List"],"sub_categories":["Follow-up Papers"],"readme":"# COCO-ReM (COCO with Refined Masks)\n\n[![Framework: PyTorch](https://img.shields.io/badge/Framework-PyTorch-orange.svg)](https://pytorch.org) [![HuggingFace Datasets](https://img.shields.io/badge/%F0%9F%A4%97-HuggingFace_Datasets-cyan.svg\n)](https://huggingface.co/datasets/kdexd/coco-rem)\n\n[Shweta Singh](https://www.linkedin.com/in/shweta-singh-460154284/)\u003csup\u003e\u0026dagger;\u003c/sup\u003e, [Aayan Yadav](https://www.linkedin.com/in/aayanyadav09/)\u003csup\u003e\u0026dagger;\u003c/sup\u003e, [Jitesh Jain](https://praeclarumjj3.github.io/), [Humphrey Shi](https://www.humphreyshi.com/home), [Justin Johnson](https://web.eecs.umich.edu/~justincj/), [Karan Desai](https://kdexd.xyz/) \n\n\u003csup\u003e\u0026dagger;\u003c/sup\u003e Equal Contribution\n\n[[`arxiv`](https://arxiv.org/abs/2403.18819)] [[`Dataset Website`](https://cocorem.xyz)]\n\n![Random examples from COCO-ReM](./images/coco_rem_example_2.jpg)\n\nIntroducing COCO-ReM, a set of high-quality instance annotations for COCO images.\nCOCO-ReM improves on imperfections prevailing in COCO-2017 such as coarse mask boundaries, non-exhaustive annotations,\ninconsistent handling of occlusions, and duplicate masks.\nMasks in COCO-ReM have a visibly better quality than COCO-2017, as shown below.\n\n![COCO and COCO-ReM](./images/coco_rem_example_1.jpg)\n\n## Contents\n\n1. [News](#news)\n2. [Setup Instructions](#setup-instructions)\n3. [Download COCO-ReM](#download-coco-rem)\n4. [Mask Visualization](#mask-visualization)\n5. [Evaluation using COCO-ReM](#evaluation-using-coco-rem)\n6. [Training with COCO-ReM](#training-with-coco-rem)\n7. [Annotation Pipeline](#annotation-pipeline)\n    - [Stage 1: Mask Boundary Refinement (automatic step)](#stage-1-mask-boundary-refinement)\n    - [Stage 2: Exhaustive Instance Annotation (automatic step)](#stage-2-exhaustive-instance-annotation)\n    - [Stage 3: Correction of Labeling Errors](#stage-3-correction-of-labeling-errors)\n8. [Citation](#citation)\n\n## News\n\n- **[July 7, 2024]**: Dataset now available on [**HuggingFace**](https://huggingface.co/datasets/kdexd/coco-rem) and [**code**](https://github.com/kdexd/coco-rem) is public!\n- **[July 1, 2024]**: COCO-ReM is accepted to ECCV 2024!\n- **[March 27, 2024]**: [**Dataset website**](https://cocorem.xyz) and [**arXiv preprint**](https://arxiv.org/abs/2403.18819) are public!\n\n## Setup Instructions\n\nClone the repository, create a conda environment, and install all dependencies as follows:\n\n```bash\ngit clone https://github.com/kdexd/coco-rem.git \u0026\u0026 cd coco-rem\nconda create -n coco_rem python=3.10\nconda activate coco_rem\n```\n\nInstall PyTorch and `torchvision` following the instructions on [pytorch.org](https://pytorch.org).\nInstall Detectron2, [instructions are available here](https://github.com/facebookresearch/detectron2/blob/main/INSTALL.md).\nThen, install the dependencies:\n\n```bash\npip install -r requirements.txt\npip install git+https://github.com/facebookresearch/segment-anything.git\npip install git+https://github.com/bowenc0221/boundary-iou-api.git\n\npython setup.py develop\n```\n\n## Download COCO-ReM\n\nCOCO-ReM is hosted on Huggingface Datasets at [@kdexd/coco-rem](https://huggingface.co/datasets/kdexd/coco-rem).\nDownload the annotation files:\n\n```\nfor name in trainrem valrem; do\n    wget https://huggingface.co/datasets/kdexd/coco-rem/resolve/main/instances_$name.json.zip\n    unzip instances_$name.json.zip\ndone\n```\n\n**Dataset organization:** COCO and COCO-ReM and must be organized inside `datasets` directory as follows.\n\n```\n$PROJECT_ROOT/datasets\n    — coco/\n        — train2017/         # Contains 118287 train images (.jpg files).\n        — val2017/           # Contains 5000 val images (.jpg files).\n        — annotations/\n            — instances_train2017.json\n            — instances_val2017.json\n    - coco_rem/\n            - instances_trainrem.json\n            - instances_valrem.json\n    -lvis\n            - lvis_v1_val.json\n            - lvis_v1_train.json\n```\n\n-----\n\n## Mask Visualization\n\nWe include a lightweight script to quickly visualize masks of COCO-ReM and COCO-2017,\nboth validation and training sets. For example, run the following command to visualize\nthe masks for COCO-ReM validation set:\n\n```bash\npython scripts/visualize_coco.py \\\n    --input-json datasets/coco_rem/instances_valrem.json \\\n    --image-dir datasets/coco/val2017 \\\n    --output visualization_output\n```\n\nRead the documentation (`python scripts/visualize_coco.py --help`) for details about other arguments.\n\n-----\n\n## Evaluation using COCO-ReM\n\nWe support evaluation of all fifty object detectors available in the paper.\nFirst, run `python checkpoints/download.py` to download all the pre-trained models\nfrom their official repositories and save them in `checkpoints/pretrained_weights`.\n\nFor example, to evaluate a [Mask R-CNN ViTDet-B model](https://arxiv.org/abs/2203.16527) using 8 GPUs\nand calculate average precision (AP) metrics, run the following command:\n\n```bash\npython scripts/train_net.py --num-gpus 8 --eval-only \\\n    --config coco_rem/configs/vitdet/mask_rcnn_vitdet_b_100ep.py \\\n    train.init_checkpoint=checkpoints/pretrained_weights/vitdet/mask_rcnn_vitdet_b_100ep.pkl \\\n    dataloader.test.dataset.names=coco_rem_val \\\n    train.output_dir=evaluation_results\n```\n\n## Training with COCO-ReM\n\nWe also support training ViTDet baselines on COCO-ReM using the Detectron2 library.\nRun the following command to train using 8 GPUs (with at least 32GB memory):\n\n```bash\npython scripts/train_net.py --num-gpus 8 \\\n    --config coco_rem/configs/vitdet/mask_rcnn_vitdet_b_100ep.py \\\n    dataloader.train.dataset.names=coco_rem_train \\\n    dataloader.test.dataset.names=coco_rem_val \\\n    train.output_dir=training_output \\\n    dataloader.train.total_batch_size=16 train.grad_accum_steps=4\n```\n\nFor GPUs with less memory, update the parameters in the last line above:\nthe batch size can be halved and gradient accumulation steps can be doubled, for same results.\n\n## Annotation Pipeline\n\n\u003ca id=\"stage-1-mask-boundary-refinement\"\u003e\u003c/a\u003e\n### Stage 1: Mask Boundary Refinement (automatic step)\n\nDownload checkpoint for SAM from [segment-anything repository](https://github.com/facebookresearch/segment-anything) and place it in `checkpoint` folder.\n\nRun the following command to refine the boundaries of validation set masks using 8 GPUs:\n\n```bash\npython scripts/refine_boundaries.py \\\n    --input-json datasets/coco/annotations/instances_val2017.json \\\n    --image-dir datasets/coco/val2017 \\\n    --num-gpus 8 \\\n    --output datasets/intermediate/cocoval_boundary_refined.json\n```\n\nRead the documentation (`python scripts/refine_boundaries.py --help`) for details about other arguments.\n\nUse default values for other optional arguments to follow the strategy used in [paper](https://arxiv.org/abs/2403.18819). \n\nDo this stage for both COCO and LVIS datasets before the merging stage.\n\n\u003ca id=\"stage-2-exhaustive-instance-annotation\"\u003e\u003c/a\u003e\n\n### Stage 2: Exhaustive Instance Annotation (automatic step)\n\nRun the following command to merge LVIS annotations for validation set of COCO using the strategy described in  [paper](https://arxiv.org/abs/2403.18819):\n\n```\npython scripts/merge_instances.py \\\n    --coco-json datasets/intermediate/cocoval_boundary_refined.json \\\n    --lvis-json datasets/intermediate/lvistrain_boundary_refined.json datasets/intermediate/lvisval_boundary_refined.json \\\n    --split val \\\n    --output datasets/intermediate/cocoval_lvis_merged.json\n```\nRead the documentation (`python scripts/merge_instances.py --help`) for details about above arguments.\n\nMerging handpicked `(image,category)` non exhaustive instances from LVIS in validation set is done in the script of next stage.\n\n\u003ca id=\"stage-3-correction-of-labeling-errors\"\u003e\u003c/a\u003e\n\n### Stage 3: Correction of Labeling Errors\n\nThis stage is done only for validation set.\n\n```\npython scripts/correct_labeling_errors.py \\\n    --input datasets/intermediate/cocoval_lvis_merged.json \\\n    --output datasets/cocoval_refined.json\n```\n**Note**: For the above json to be COCO-ReM we also have to perform the manual parts of Stage 1 and Stage 2.\n\n## Citation\n\nIf you found COCO-ReM useful in your research, please consider starring ⭐ us on GitHub and citing 📚 us in your research!\n\n```bibtex\n@inproceedings{cocorem,\n  title={Benchmarking Object Detectors with COCO: A New Path Forward},\n  author={Singh, Shweta and Yadav, Aayan and Jain, Jitesh and Shi, Humphrey and Johnson, Justin and Desai, Karan},\n  journal={ECCV},\n  year={2024}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkdexd%2Fcoco-rem","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkdexd%2Fcoco-rem","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkdexd%2Fcoco-rem/lists"}