{"id":19216992,"url":"https://github.com/hustvl/mimdet","last_synced_at":"2025-04-06T09:08:19.572Z","repository":{"id":40060049,"uuid":"476640733","full_name":"hustvl/MIMDet","owner":"hustvl","description":"[ICCV 2023] You Only Look at One Partial Sequence","archived":false,"fork":false,"pushed_at":"2023-10-21T03:43:34.000Z","size":564,"stargazers_count":340,"open_issues_count":8,"forks_count":30,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-03-30T07:10:09.616Z","etag":null,"topics":["computer-vision","instance-segmentation","mae","masked-image-modeling","object-detection","transformer","vision-transformer"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2204.02964","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hustvl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-01T08:40:01.000Z","updated_at":"2025-03-27T06:42:40.000Z","dependencies_parsed_at":"2024-06-19T03:02:28.521Z","dependency_job_id":"73c58c6c-6c98-4342-8ff4-d953f10cb802","html_url":"https://github.com/hustvl/MIMDet","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hustvl%2FMIMDet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hustvl%2FMIMDet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hustvl%2FMIMDet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hustvl%2FMIMDet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hustvl","download_url":"https://codeload.github.com/hustvl/MIMDet/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247457801,"owners_count":20941906,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","instance-segmentation","mae","masked-image-modeling","object-detection","transformer","vision-transformer"],"created_at":"2024-11-09T14:19:46.295Z","updated_at":"2025-04-06T09:08:19.550Z","avatar_url":"https://github.com/hustvl.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003ch1\u003eMIMDet \u0026#127917;\u003c/h1\u003e\n\u003ch3\u003eUnleashing Vanilla Vision Transformer\nwith Masked Image Modeling for Object Detection\u003c/h3\u003e\n\n\n[Yuxin Fang](https://bit.ly/YuxinFang_GoogleScholar)\u003csup\u003e1\u003c/sup\u003e \\*, [Shusheng Yang](https://scholar.google.com/citations?user=v6dmW5cntoMC\u0026hl=en)\u003csup\u003e1\u003c/sup\u003e \\*, [Shijie Wang](https://github.com/simonJJJ)\u003csup\u003e1\u003c/sup\u003e \\*, [Yixiao Ge](https://geyixiao.com/)\u003csup\u003e2\u003c/sup\u003e, [Ying Shan](https://scholar.google.com/citations?user=4oXBp9UAAAAJ\u0026hl=en)\u003csup\u003e2\u003c/sup\u003e, [Xinggang Wang](https://xinggangw.info/)\u003csup\u003e1 :email:\u003c/sup\u003e,\n \n\u003csup\u003e1\u003c/sup\u003e [School of EIC, HUST](http://eic.hust.edu.cn/English/Home.htm), \u003csup\u003e2\u003c/sup\u003e [ARC Lab, Tencent PCG](https://arc.tencent.com/en/index).\n\n(\\*) equal contribution, (\u003csup\u003e:email:\u003c/sup\u003e) corresponding author.\n\n**ICCV 2023** [[paper](https://openaccess.thecvf.com/content/ICCV2023/papers/Fang_Unleashing_Vanilla_Vision_Transformer_with_Masked_Image_Modeling_for_Object_ICCV_2023_paper.pdf)]\n\n\u003c/div\u003e\n\n# \n\n### News\n\n* **`19 May, 2022`:** We update our [preprint](https://arxiv.org/abs/2204.02964v2) with stronger results and more analysis. Code \u0026 models are also updated in the [`main`](https://github.com/hustvl/MIMDet/tree/main) branch. For our previous results (code \u0026 models), please refer to the [`v1.0.0`](https://github.com/hustvl/MIMDet/tree/v1.0.0) branch.\n\n* **`6 Apr, 2022`:** Code \u0026 models are released!\n\n## Introduction\n\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"MIMDet.png\" width=80%\u003e\n\u003c/p\u003e\n\nThis repo provides code and pretrained models for **MIMDet** (**M**asked **I**mage **M**odeling for **Det**ection).\n* MIMDet is a simple framekwork that enables a MIM pretrained vanilla ViT to perform high-performance object-level understanding, e.g, object detection and instance segmentation.\n* In MIMDet, a MIM pre-trained vanilla ViT encoder can work surprisingly well in the challenging object-level recognition scenario even with randomly sampled *partial* observations, e.g., only 25%~50% of the input embeddings.\n* In order to construct multi-scale representations for object detection, a *randomly initialized* compact convolutional stem supplants the pre-trained large kernel patchify stem, and its intermediate features can naturally serve as the higher resolution inputs of a feature pyramid without upsampling. While the pre-trained ViT is only regarded as the third-stage of our detector's backbone instead of the whole feature extractor, resulting in a ConvNet-ViT *hybrid* architecture.\n* MIMDet w/ ViT-Base \u0026 Mask R-CNN FPN obtains **51.7 box AP** and **46.2 mask AP** on COCO. With ViT-L, MIMDet achieves **54.3 box AP** and **48.2 mask AP**.\n* We also provide an unofficial implementation of [Benchmarking Detection Transfer Learning with Vision Transformers](https://arxiv.org/abs/2111.11429) that successfully reproduces its reported results.\n\n## Models and Main Results\n\n### Mask R-CNN\n| \u003csub\u003eModel | \u003csub\u003eSample Ratio | \u003csub\u003eSchedule | \u003csub\u003eAug | \u003csub\u003eBox AP | \u003csub\u003eMask AP | \u003csub\u003e#params | \u003csub\u003econfig | \u003csub\u003emodel / log |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| \u003csub\u003e**MIMDet-ViT-B** | \u003csub\u003e0.5 | \u003csub\u003e3x | \u003csub\u003e[480-800, 1333] w/crop | \u003csub\u003e51.7 | \u003csub\u003e46.2 | \u003csub\u003e127.96M | \u003csub\u003e[config](configs/mimdet/mimdet_vit_base_mask_rcnn_fpn_sr_0p5_800_1333_4xdec_coco_3x.py) | \u003csub\u003e[model](https://github.com/hustvl/Storage/releases/download/v1.2.0/mimdet_vit_base_mask_rcnn_fpn_sr_0p5_800_1333_4xdec_coco_3x.pth) / [log](https://github.com/hustvl/Storage/releases/download/v1.2.0/mimdet_vit_base_mask_rcnn_fpn_sr_0p5_800_1333_4xdec_coco_3x.json) |\n| \u003csub\u003e**MIMDet-ViT-L** | \u003csub\u003e0.5 | \u003csub\u003e3x | \u003csub\u003e[480-800, 1333] w/crop | \u003csub\u003e54.3 | \u003csub\u003e48.2 | \u003csub\u003e349.33M | \u003csub\u003e[config](configs/mimdet/mimdet_vit_large_mask_rcnn_fpn_sr_0p5_800_1333_4xdec_coco_3x.py) | \u003csub\u003e[model](https://github.com/hustvl/Storage/releases/download/v1.2.0/mimdet_vit_large_mask_rcnn_fpn_sr_0p5_800_1333_4xdec_coco_3x.pth) / [log](https://github.com/hustvl/Storage/releases/download/v1.2.0/mimdet_vit_large_mask_rcnn_fpn_sr_0p5_800_1333_4xdec_coco_3x.json) |\n| \u003csub\u003eBenchmarking-ViT-B | \u003csub\u003e- | \u003csub\u003e25ep | \u003csub\u003e[1024, 1024] LSJ(0.1-2) | \u003csub\u003e48.0 | \u003csub\u003e43.0 | \u003csub\u003e118.67M | \u003csub\u003e[config](configs/benchmarking/benchmarking_mask_rcnn_base_FPN_25ep_LSJ_mae.py) | \u003csub\u003e[model](https://github.com/hustvl/storage/releases/download/v1.0.0/benchmarking_mask_rcnn_base_FPN_25ep_LSJ_mae.pth) / [log](https://github.com/hustvl/Storage/releases/download/v1.0.1/benchmarking_mask_rcnn_base_FPN_25ep_LSJ_mae.json) |\n| \u003csub\u003eBenchmarking-ViT-B | \u003csub\u003e- | \u003csub\u003e50ep | \u003csub\u003e[1024, 1024] LSJ(0.1-2) | \u003csub\u003e50.2 | \u003csub\u003e44.9 | \u003csub\u003e118.67M | \u003csub\u003e[config](configs/benchmarking/benchmarking_mask_rcnn_base_FPN_50ep_LSJ_mae.py) | \u003csub\u003e[model](https://github.com/hustvl/storage/releases/download/v1.0.0/benchmarking_mask_rcnn_base_FPN_50ep_LSJ_mae.pth) / [log](https://github.com/hustvl/Storage/releases/download/v1.0.1/benchmarking_mask_rcnn_base_FPN_50ep_LSJ_mae.json) |\n| \u003csub\u003eBenchmarking-ViT-B |\u003csub\u003e- | \u003csub\u003e100ep | \u003csub\u003e[1024, 1024] LSJ(0.1-2) | \u003csub\u003e50.4 | \u003csub\u003e44.9 | \u003csub\u003e118.67M | \u003csub\u003e[config](configs/benchmarking/benchmarking_mask_rcnn_base_FPN_100ep_LSJ_mae.py) | \u003csub\u003e[model](https://github.com/hustvl/storage/releases/download/v1.0.0/benchmarking_mask_rcnn_base_FPN_100ep_LSJ_mae.pth) / [log](https://github.com/hustvl/Storage/releases/download/v1.0.1/benchmarking_mask_rcnn_base_FPN_100ep_LSJ_mae.json) |\n\n**Notes**:\n\n- The Box AP \u0026 Mask AP in the table above is obtained w/ sample ratio = 1.0, which is higher than the training sample ratio (0.25 or 0.5). Our MIMDet can benefit from lower sample ratio during training for better efficiency, as well as higher sample ratio during inference for better accuracy. Please refer to our paper for detailed analysis.\n- Benchmarking-ViT-B is an unofficial implementation of [Benchmarking Detection Transfer Learning with Vision Transformers](https://arxiv.org/abs/2111.11429).\n\n## Installation\n\n### Prerequisites\n* Linux\n* Python 3.7+\n* CUDA 10.2+\n* GCC 5+\n\n### Prepare\n\n- Clone\n```\ngit clone https://github.com/hustvl/MIMDet.git\ncd MIMDet\n```\n\n\n- Create a conda virtual environment and activate it:\n```\nconda create -n mimdet python=3.9\nconda activate mimdet\n```\n\n* Install `torch==1.9.0` and `torchvision==0.10.0`\n* Install [`Detectron2==0.6`](https://github.com/facebookresearch/detectron2), follow [d2 doc](https://detectron2.readthedocs.io/tutorials/install.html).\n* Install [`timm==0.4.12`](https://github.com/rwightman/pytorch-image-models), follow [timm doc](https://fastai.github.io/timmdocs/).\n* Install [`einops`](https://github.com/arogozhnikov/einops), follow [einops repo](https://github.com/arogozhnikov/einops#installation--).\n* Prepare [`COCO`](https://cocodataset.org/#home) dataset, follow [d2 doc](https://detectron2.readthedocs.io/en/latest/tutorials/builtin_datasets.html).\n\n### Dataset\n\nMIMDet is built upon ```detectron2```, so please organize dataset directory in detectron2's manner. We refer users to [```detectron2```](https://github.com/facebookresearch/detectron2) for detailed instructions. The overall hierachical structure is illustrated as following:\n```\nMIMDet\n├── datasets\n│   ├── coco\n│   │   ├── annotations\n│   │   ├── train2017\n│   │   ├── val2017\n│   │   ├── test2017\n│   ├── ...\n├── ...\n```\n\n\n## Training\n\nDownload the ***full*** MAE pretrained (including the decoder) [ViT-B Model](https://dl.fbaipublicfiles.com/mae/pretrain/mae_pretrain_vit_base_full.pth) and [ViT-L Model](https://dl.fbaipublicfiles.com/mae/pretrain/mae_pretrain_vit_large_full.pth) checkpoint. See [MAE repo-issues-8](https://github.com/facebookresearch/mae/issues/8).\n```\n# single-machine training\npython lazyconfig_train_net.py --config-file \u003cCONFIG_FILE\u003e --num-gpus \u003cGPU_NUM\u003e mae_checkpoint.path=\u003cMAE_MODEL_PATH\u003e\n\n# multi-machine training\npython lazyconfig_train_net.py --config-file \u003cCONFIG_FILE\u003e --num-gpus \u003cGPU_NUM\u003e --num-machines \u003cMACHINE_NUM\u003e --master_addr \u003cMASTER_ADDR\u003e --master_port \u003cMASTER_PORT\u003e mae_checkpoint.path=\u003cMAE_MODEL_PATH\u003e\n```\n\n## Inference\n\n```\n# inference\npython lazyconfig_train_net.py --config-file \u003cCONFIG_FILE\u003e --num-gpus \u003cGPU_NUM\u003e --eval-only train.init_checkpoint=\u003cMODEL_PATH\u003e\n\n# inference with 100% sample ratio (please refer to our paper for detailed analysis)\npython lazyconfig_train_net.py --config-file \u003cCONFIG_FILE\u003e --num-gpus \u003cGPU_NUM\u003e --eval-only train.init_checkpoint=\u003cMODEL_PATH\u003e model.backbone.bottom_up.sample_ratio=1.0\n```\n\n## Acknowledgement\nThis project is based on [MAE](https://github.com/facebookresearch/mae), [Detectron2](https://github.com/facebookresearch/detectron2) and [timm](https://github.com/rwightman/pytorch-image-models). Thanks for their wonderful works.\n\n## License\n\nMIMDet is released under the [MIT License](LICENSE).\n\n## Citation\n\nIf you find our paper and code useful in your research, please consider giving a star :star: and citation :pencil: :)\n\n```BibTeX\n@article{MIMDet,\n  title={Unleashing Vanilla Vision Transformer with Masked Image Modeling for Object Detection},\n  author={Fang, Yuxin and Yang, Shusheng and Wang, Shijie and Ge, Yixiao and Shan, Ying and Wang, Xinggang},\n  journal={arXiv preprint arXiv:2204.02964},\n  year={2022}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhustvl%2Fmimdet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhustvl%2Fmimdet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhustvl%2Fmimdet/lists"}