{"id":13738068,"url":"https://github.com/clovaai/wsolevaluation","last_synced_at":"2025-04-06T10:13:29.390Z","repository":{"id":37732026,"uuid":"235301094","full_name":"clovaai/wsolevaluation","owner":"clovaai","description":"Evaluating Weakly Supervised Object Localization Methods Right (CVPR 2020)","archived":false,"fork":false,"pushed_at":"2022-09-20T15:31:32.000Z","size":12409,"stargazers_count":334,"open_issues_count":11,"forks_count":57,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-03-30T09:06:38.307Z","etag":null,"topics":["cvpr2020","dataset-contribution","evaluation-protocol","localization","wsol-benchmark","wsol-evaluation","wsol-methods","wsol-task","wsol-training"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/clovaai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-01-21T09:26:09.000Z","updated_at":"2025-03-12T02:23:39.000Z","dependencies_parsed_at":"2023-01-17T17:00:48.449Z","dependency_job_id":null,"html_url":"https://github.com/clovaai/wsolevaluation","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clovaai%2Fwsolevaluation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clovaai%2Fwsolevaluation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clovaai%2Fwsolevaluation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clovaai%2Fwsolevaluation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/clovaai","download_url":"https://codeload.github.com/clovaai/wsolevaluation/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247464223,"owners_count":20942970,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cvpr2020","dataset-contribution","evaluation-protocol","localization","wsol-benchmark","wsol-evaluation","wsol-methods","wsol-task","wsol-training"],"created_at":"2024-08-03T03:02:10.343Z","updated_at":"2025-04-06T10:13:29.365Z","avatar_url":"https://github.com/clovaai.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"## Evaluating Weakly Supervised Object Localization Methods Right (CVPR 2020)\n\n[CVPR 2020 paper](https://arxiv.org/abs/2001.07437) | [TPAMI paper](https://arxiv.org/abs/2007.04178)\n\nJunsuk Choe\u003csup\u003e1,3*\u003c/sup\u003e, Seong Joon Oh\u003csup\u003e2*\u003c/sup\u003e, Seungho Lee\u003csup\u003e1\u003c/sup\u003e, \nSanghyuk Chun\u003csup\u003e3\u003c/sup\u003e, Zeynep Akata\u003csup\u003e4\u003c/sup\u003e, Hyunjung Shim\u003csup\u003e1\u003c/sup\u003e  \n\u003csub\u003e\\* Equal contribution\u003c/sub\u003e\n\n\n\u003csup\u003e1\u003c/sup\u003e \u003csub\u003eSchool of Integrated Technology, Yonsei University\u003c/sub\u003e  \n\u003csup\u003e2\u003c/sup\u003e \u003csub\u003eClova AI Research, LINE Plus Corp.\u003c/sub\u003e\n\u003csup\u003e3\u003c/sup\u003e \u003csub\u003eClova AI Research, NAVER Corp.\u003c/sub\u003e\n\u003csup\u003e4\u003c/sup\u003e \u003csub\u003eUniversity of T\u0026uuml;bingen\u003c/sub\u003e\n\nWeakly-supervised object localization (WSOL) has gained popularity over the \nlast years for its promise to train localization models with only image-level \nlabels. Since the seminal WSOL work of class activation mapping (CAM), \nthe field has focused on how to expand the attention regions to cover objects \nmore broadly and localize them better. However, these strategies rely on \nfull localization supervision to validate hyperparameters and for model \nselection, which is in principle prohibited under the WSOL setup. In this paper,\n we argue that WSOL task is ill-posed with only image-level labels, and \n propose a new evaluation protocol where full supervision is limited to only \n a small held-out set not overlapping with the test set. We observe that, \n under our protocol, the five most recent WSOL methods have not made a major \n improvement over the CAM baseline. Moreover, we report that existing WSOL \n methods have not reached the few-shot learning baseline, where the \n full-supervision at validation time is used for model training instead. \n Based on our findings, we discuss some future directions for WSOL.\n\n\u003cimg src=\"teaser.png\" width=\"60%\" height=\"60%\" title=\"\" alt=\"RubberDuck\"\u003e\u003c/img\u003e\n\n__Overview of WSOL performances 2016-2019.__ Above image shows that recent improvements in WSOL are \nillusory due to (1) different amount of implicit full supervision through \nvalidation and (2) a fixed score-map threshold to generate object boxes. Under \nour evaluation protocol with the same validation set sizes and oracle threshold \nfor each method, CAM is still the best. In fact, our few-shot learning baseline, \ni.e., using the validation supervision (10 samples/class) at training time, \noutperforms existing WSOL methods.\n\n## Updates\n\n- __9 Jul, 2020__: [Journal submission](https://arxiv.org/abs/2007.04178) available.\n- __27 Mar, 2020__: [New WSOL evaluations](#5-library-of-wsol-methods) with `MaxBoxAccV2` are added.\n- __28 Feb, 2020__: [New box evaluation](#improved-box-evaluation) (`MaxBoxAccV2`) is available.\n- __22 Jan, 2020__: Initial upload.\n\n\n## Table of contents \n\n* [1. Our dataset contribution](#1-our-dataset-contribution)\n    + [The dataset splits](#the-dataset-splits)\n* [2. Dataset downloading and license](#2-dataset-downloading-and-license)\n    + [ImageNetV2](#imagenetv2)\n    + [CUBV2](#cubv2)\n    + [OpenImages30k](#openimages30k)\n    + [Dataset statistics](#dataset-statistics)\n    + [Licenses](#licenses)\n* [3. Code dependencies](#3-code-dependencies)\n* [4. WSOL evaluation](#4-wsol-evaluation)\n    + [Prepare evaluation data](#prepare-evaluation-data)\n    + [Prepare heatmaps to evaluate](#prepare-heatmaps-to-evaluate)\n    + [Evaluate your heatmaps](#evaluate-your-heatmaps)\n    + [Testing the evaluation code](#testing-the-evaluation-code)\n* [5. Library of WSOL methods](#5-library-of-wsol-methods)\n* [6. WSOL training and evaluation](#6-wsol-training-and-evaluation)\n    + [Prepare train+eval datasets](#prepare-traineval-datasets)\n        - [ImageNet](#imagenet)\n        - [CUB](#cub)\n        - [OpenImages](#openimages)\n    + [Run train+eval](#run-traineval)\n    + [Improved box evaluation](#improved-box-evaluation)\n* [7. Code license](#7-code-license)\n* [8. How to cite](#8-how-to-cite)\n\n## 1. Our dataset contribution\n\nWSOL is an ill-posed problem when only image-level labels are available (see paper for an argument).\nTo be able to solve the WSOL task, certain amount of full supervision is inevitable, and prior WSOL approaches have utilized\ndifferent amount of implicit and explicit full supervision (usually through validation).\nWe propose to use a fixed amount of full supervision per method by carefully designing validation splits (called `train-fullsup` in the paper), such that different methods use the same amount of localization-labelled validation split.\n\nIn this section, we explain how each dataset is split, and introduce our data contributions (image collections and new annotations) on the way.\n\n### The dataset splits\n\nsplit           | ImageNet               | CUB                        | OpenImages\n----------------|------------------------|----------------------------|---------------------------\n`train-weaksup` | ImageNet \"train\"       | CUB-200-2011 \"train\"       | OpenImages30k \"train\"  ![we_curated data](https://img.shields.io/badge/we_curated-data-lightgray.svg)\n`train-fullsup` | ImageNetV2  ![we_collected annotations](https://img.shields.io/badge/we_collected-annotations-lightgray.svg)            | CUBV2  ![we_collected images_and_annotations](https://img.shields.io/badge/we_collected-images_and_annotations-lightgray.svg)  | OpenImages30k \"val\"  ![we_curated data](https://img.shields.io/badge/we_curated-data-lightgray.svg)\n`test`          | ImageNet \"val\"         | CUB-200-2011 \"test\"        | OpenImages30k \"test\"  ![we_curated data](https://img.shields.io/badge/we_curated-data-lightgray.svg)\n\n\nWe propose three disjoint splits for every dataset: `train-weaksup`, \n`train-fullsup`, and `test`. The `train-weaksup` contains images with weak \nsupervision (the image-level labels). The `train-fullsup` contains images with \nfull supervision (either bounding box or binary mask). It is left as freedom for \nthe user to utilize it for hyperparameter search, model selection, ablative \nstudies, or even model fitting. The `test` split contains images with full \nsupervision; it must be used only for the final performance report. For \nexample, checking the `test` results multiple times with different model \nconfigurations violates the protocol as the learner implicitly uses more full \nsupervision than allowed. The splits and their roles are more extensively explained\nin the paper.\n\n- ImageNet\n  - \"train\" and \"val\" splits of original [ImageNet](http://www.image-net.org/) \n  are treated as our `train-weaksup` and `test`. \n  - [ImageNetV2](https://github.com/modestyachts/ImageNetV2) is treated as our \n  `train-fullsup`. Note that we have annotated bounding boxes on ImageNetV2. \n- CUB\n  - \"train\" and \"test\" splits of original \n  [CUB-200-2011](http://www.vision.caltech.edu/visipedia/CUB-200-2011.html) are \n  treated as our `train-weaksup` and `test`.\n  - We contribute images and annotations that are similar as the original CUB, \n  namely CUBV2.\n- OpenImages\n  - We curate the existing [OpenImagesV5](https://storage.googleapis.com/openimages/web/index.html) \n  for the task of WSOL.\n  - We have randomly selected images from the original \"train\", \"val\", and \n  \"test\" splits of the instance segmentation subset.\n\n## 2. Dataset downloading and license\n\nFor original ImageNet and CUB datasets, please follow the common procedure to \ndownload the datasets. In this section, we only explain how to obtain the less used \n(or never used before) datasets. We also provide the license status for each dataset.\nThis section is for those who are interested in the full data for each dataset.\nIf the aim is to utilize the data for WSOL evaluation and/or training, please follow the links below:\n\n* [Evaluation only](#prepare-evaluation-data)\n* [Training + evaluation](#prepare-traineval-datasets)\n\n#### ImageNetV2\n[Download images](https://s3-us-west-2.amazonaws.com/imagenetv2public/imagenetv2-threshold0.7.tar.gz)\n\nWe utilize 10,000 images in the `Threshold0.7` split of \n[ImageNetV2](https://github.com/modestyachts/ImageNetV2) for our `train-fullsup` \nsplit. We have annotated bounding boxes on those images. \nBox labels exist in [here](release/) and are licensed by NAVERCorp. under\n[Attribution 2.0 Generic (CC-BY-2.0)](https://creativecommons.org/licenses/by/2.0/).\n\n#### CUBV2\n[Download images](https://drive.google.com/open?id=1U6cwKHS65wayT9FFvoLIA8cn1k0Ot2M1) \n\nWe have collected and annotated CUBV2 on our own as the `train-fullsup` split.\nWe have ensured that the data distribution\nfollows the original CUB dataset and there is no duplicate image.\nWe have collected 5 images per class \n(1,000 images total) from [Flickr](https://www.flickr.com/). \nBox labels and license files of all images exist in [here](release/). \nBoth class and box labels are licensed by NAVERCorp under\n[Attribution 2.0 Generic (CC-BY-2.0)](https://creativecommons.org/licenses/by/2.0/).\n\n#### OpenImages30k\n[Download images](https://drive.google.com/open?id=1oOb4WQ-lb8SYppHEg3lWnpHk1X8WVO4e)   \n[Download segmentation masks](https://drive.google.com/open?id=1eu1YvcZlsEalhXTS_5Ni5tkImCliIPie)\n\nThe WSOL community has relied on ImageNet and CUB datasets at least for the last three years.\nIt is perhaps time for us to move on. We provide a WSOL benchmark based on the OpenImages30k\ndataset to provide a new perspective on the generalizability of \nWSOL methods in the past and future. To make it suitable for the WSOL task, \nwe use 100 classes to ensure the minimum number of single-class samples for \neach class. We have randomly selected 29,819, 2,500, and 5,000 images from the \noriginal \"train\", \"val\", and \"test\" splits of \n[OpenImagesV5](https://storage.googleapis.com/openimages/web/index.html).\nCorresponding metadata can be found in [here](metadata/OpenImages/).\nThe annotations are licensed by Google LLC under \n[Attribution 4.0 International (CC-BY-4.0)](https://creativecommons.org/licenses/by/4.0/).\nThe images are listed as having a \n[Attribution 2.0 Generic (CC-BY-2.0)](https://creativecommons.org/licenses/by/2.0/).\n\n#### Dataset statistics\nBelow tables summarizes dataset statistics of each split.\n\n\\# images\\/classes | ImageNet  \u003csub\u003e1,000 classes\u003c/sub\u003e | CUB  \u003csub\u003e200 classes\u003c/sub\u003e | OpenImages  \u003csub\u003e100 classes\u003c/sub\u003e\n-------------------|----------|-----|------------\n`train-weaksup`    | ~1,200   | ~30 | ~300\n`train-fullsup`    | 10       | ~5  | 25\n`test`             | 10       | ~29 | 50\n\n#### Licenses \n\nThe licenses corresponding to our dataset contribution are summarized as follows\n\nDataset    | Images                                   | Class Annotations | Localization Annotations\n-----------|------------------------------------------|-------------------|----------------\nImageNetV2 | See the [original Github](https://github.com/modestyachts/ImageNetV2) |  See the [original Github](https://github.com/modestyachts/ImageNetV2)  | CC-BY-2.0 NaverCorp.\nCUBV2      | Follows original image licenses. See [here](release/CUBV2_image_licenses.txt). | CC-BY-2.0 NaverCorp. | CC-BY-2.0 NaverCorp.\nOpenImages | CC-BY-2.0 (Follows original image licenses. See [here](https://storage.googleapis.com/openimages/web/factsfigures.html)) | CC-BY-4.0 Google LLC | CC-BY-4.0 Google LLC\n\nDetailed license files are summarized in the [release](release) directory.  \n\n**Note**: At the time of collection, images were marked as being licensed under \nthe following licenses:\n```\nAttribution-NonCommercial License\nAttribution License\nPublic Domain Dedication (CC0)\nPublic Domain Mark\n```\nHowever, we make no representations or warranties regarding the license status \nof each image. You should verify the license for each image yourself.\n\n\n## 3. Code dependencies\n\nBoth the [evaluation-only](#4-wsol-evaluation) and [eval+train](#6-wsol-training-and-evaluation) scripts require only the following libraries: \n* [OpenCV](https://opencv.org/)\n* [PyTorch](https://pytorch.org/)\n* [munch](https://github.com/Infinidat/munch)\n\n`pip freeze` returns the version information as below:\n```\nmunch==2.5.0\nnumpy==1.18.1\nopencv-python==4.1.2.30\nPillow==7.0.0\nsix==1.14.0\ntorch==1.4.0\ntorchvision==0.5.0\n```\n\n## 4. WSOL evaluation\n\nWe support evaluation of weakly-supervised object localization (WSOL) methods \non CUB, ImageNet, and OpenImages. The main script for evaluation is \n[evaluation.py](evaluation.py). We will show how to download the `train-fullsup` \n(validation) and `test` set images and localization annotations.\nAn example evaluation script will be provided. \n\n#### Prepare evaluation data\n\nWSOL evaluation data consist of images and corresponding localization ground \ntruths. On CUB and ImageNet, they are given as boxes, and on OpenImages, they \nare given as binary masks. \n\nTo prepare evaluation data, first, download ImageNet \"val\" split from \n[here](http://www.image-net.org/) and put the downloaded file on \n`dataset/ILSVRC2012_img_val.tar`.\n \nThen, run the following command\n```bash\n./dataset/prepare_evaluation_data.sh\n```\n\nThe script will download the `train-fullsup` (validation) and `test` images at \n[dataset](dataset). Metadata and box annotations already exist in this repository \nunder [metadata](metadata). OpenImages mask annotations are also downloaded by \nthe above script, and will be saved under [dataset](dataset) with the images.\n \nThe structure of image files looks like\n```\ndataset\n└── ILSVRC\n    └── val2\n        └── 0\n            ├── 0.jpeg\n            ├── 1.jpeg\n            └── ...\n        └── 1\n        └── ...\n    └── val\n        ├── ILSVRC2012_val_00000001.JPEG\n        ├── ILSVRC2012_val_00000002.JPEG\n        └── ...\n└── CUB\n    └── 001.Black_footed_Albatross\n        ├── Black_Footed_Albatross_0046_18.jpg\n        ├── Black_Footed_Albatross_0002_55.jpg\n        └── ...\n    └── 002.Laysan_Albatross\n    └── ...\n└── OpenImages\n    └── val\n        └── 0bt_c3\n            ├── 1cd9ac0169ec7df0.jpg\n            ├── 1cd9ac0169ec7df0_ignore.png\n            ├── 1cd9ac0169ec7df0_m0bt_c3_6932e993.png\n            └── ...\n        └── 0bt9lr\n        └── ...\n    └── test   \n        └── 0bt_c3\n            ├── 0a51958fcd523ae4.jpg\n            ├── 0a51958fcd523ae4_ignore.png\n            ├── 0a51958fcd523ae4_m0bt_c3_41344f12.png\n            ├── 0a51958fcd523ae4_m0bt_c3_48f37c0f.png\n            └── ...\n        └── 0bt9lr\n        └── ...\n```\n\n#### Prepare heatmaps to evaluate\n\nOur WSOL evaluation evaluates heatmaps of the same width and height as the input \nimages. The evaluation script requires the heatmaps to meet the following \ncriteria:\n\n1. Heatmap file structure.\n  - Heatmaps shall be located at the user-defined `\u003cheatmap_root\u003e`.\n  - `\u003cheatmap_root\u003e` folder contains the heatmap files with the file names dictated by the `metadata/\u003cdataset\u003e/\u003csplit\u003e/image_ids.txt` files.\n  - If an `image_id` has slashes (`/`), e.g. `val2/995/0.jpeg`, then the corresponding heatmaps shall be located at the corresponding sub-directories, e.g. `\u003cheatmap_root\u003e/val2/995/0.npy`.\n\n2. Heatmap data type.\n  - Each heatmap file should be a `.npy` file that can be loaded as a numpy array with `numpy.load()`.\n  - The array shall be two-dimensional array of shape `(height, width)`, same as the input image sizes.\n  - The array shall be of type `np.float`.\n  - The array values must be between 0 and 1.\n  \n#### Evaluate your heatmaps\n\nWe support three datasets, CUB, ImageNet, and OpenImages. \n\nOn CUB and ImageNet, we evaluate the `MaxBoxAcc`, the maximal box accuracy at \nthe optimal heatmap threshold, where the box accuracy is measured by the ratio \nof images where the box generated from the heatmap overlaps with the ground \ntruth box with IoU at least `0.5`. Please see the code and paper for the full \ndetails.\n\nOn OpenImages, we evaluate the `PxAP`, pixel average precision. We generate \nthe pixel-wise precision-recall curve, and compute the area under the curve. \nPlease see the code and paper for the full details. \n\nWe present an example call to the evaluation API below:\n```bash\npython evaluation.py --scoremap_root=train_log/scoremaps/ \\\n                     --metadata_root=metadata/ \\\n                     --mask_root=dataset/ \\\n                     --dataset_name=CUB \\\n                     --split=val \\\n                     --cam_curve_interval=0.01\n```\n\nWhen `CUB` evaluation data are downloaded at `dataset` using our download script \nabove, and the corresponding heatmaps are saved under `train_log/scoremaps/`, \nthen the `MaxBoxAcc` will be evaluated as a result of this call.\n\n#### Testing the evaluation code\n\nThe test code for the evaluation modules is given at \n[evaluation_test.py](evaluation_test.py). The unit tests ensure the correctness \nof the evaluation logic, and potentially prevents unnoticed changes in the \nfunctionalities of underlying libraries (e.g. OpenCV, Numpy).\nTo run the unit test, run\n```bash\nnosetests\n```\n`pip3 install nose` may be required to install [nose](https://nose.readthedocs.io/).\n\n## 5. Library of WSOL methods\n\nWe support the training and evaluation of the following weakly-supervised object \nlocalization (WSOL) methods. Our implementation of the methods can be found in \nthe [wsol](wsol) folder. Please add your own WSOL method in the list by making \na pull request.\n\nWe provide the full training and evaluation scripts on the provided WSOL methods. \nDetails will be explained in the [next section](#6-wsol-training-and-evaluation).\n\nMethod | Paper | Original code\n-- | -- | --\nClass-Activation Mapping (CAM) | [CVPR'16](http://cnnlocalization.csail.mit.edu/Zhou_Learning_Deep_Features_CVPR_2016_paper.pdf) | [Code](https://github.com/metalbubble/CAM)\nHide-and-Seek (HaS) | [ICCV'17](http://krsingh.cs.ucdavis.edu/krishna_files/papers/hide_and_seek/my_files/iccv2017.pdf) | [Code](http://krsingh.cs.ucdavis.edu/krishna_files/papers/hide_and_seek/hide_seek.html)\nAdversarial Complementary Learning (ACoL) | [CVPR'18](http://openaccess.thecvf.com/content_cvpr_2018/papers/Zhang_Adversarial_Complementary_Learning_CVPR_2018_paper.pdf) | [Code](https://github.com/xiaomengyc/ACoL)\nSelf-Produced Guidance (SPG) | [ECCV'18](http://openaccess.thecvf.com/content_ECCV_2018/papers/Xiaolin_Zhang_Self-produced_Guidance_for_ECCV_2018_paper.pdf) | [Code](https://github.com/xiaomengyc/SPG)\nAttention-based Dropout Layer (ADL) | [CVPR'19](http://openaccess.thecvf.com/content_CVPR_2019/papers/Choe_Attention-Based_Dropout_Layer_for_Weakly_Supervised_Object_Localization_CVPR_2019_paper.pdf) | [Code](https://github.com/junsukchoe/ADL)\nCutMix | [ICCV'19](http://openaccess.thecvf.com/content_ICCV_2019/papers/Yun_CutMix_Regularization_Strategy_to_Train_Strong_Classifiers_With_Localizable_Features_ICCV_2019_paper.pdf) | [Code](https://github.com/clovaai/CutMix-PyTorch)\n\n\u003cimg src=\"main_table.png\" width=\"100%\" height=\"100%\"\u003e\u003c/img\u003e\n\n__Evaluation of WSOL methods.__ How much have WSOL methods improved upon the vanilla CAM \nmodel? `MaxBoxAccV2` and `PxAP` performances over the `test` split are shown, relative to the vanilla CAM performance. \nWe recommend the `MaxBoxAccV2` over the original box metric `MaxBoxAcc` used in the CVPR version.\nFor details, see the [latest arXiv version](https://arxiv.org/abs/2001.07437).\nHyperparameters have been optimized over the identical `train-fullsup` split for \nall WSOL methods and the FSL baseline: (10,5,5) full supervision/class for \n(ImageNet,CUB,OpenImages). Note that we evaluate the last checkpoint of each \ntraining session. More detailed results and corresponding hyperparameter sets \nare available at \n[here](https://docs.google.com/spreadsheets/d/1O4gu69FOOooPoTTtAEmFdfjs2K0EtFneYWQFk8rNqzw/edit?usp=sharing).\n\n\n## 6. WSOL training and evaluation\n\nWe describe the data preparation and training scripts for the above six prior \nWSOL methods.\n\n### Prepare train+eval datasets\n\nOur repository enables evaluation _and_ training of WSOL methods on two \ncommonly-used benchmarks, CUB and ImageNet, and our newly-introduced benchmark \nOpenImages. We describe below how to prepare those datasets.\n\n#### ImageNet\n\nBoth the original [ImageNet](http://www.image-net.org/) and \n[ImageNetV2](https://github.com/modestyachts/ImageNetV2) are required for WSOL \ntraining. Note that \"val\" split of the original ImageNet is considered as `test` \nsplit, and ImageNetV2 is used for  split (`train-fullsup`) in our framework. \n \nTo prepare ImageNet data, download ImageNet \"train\" and \"val\" splits from \n[here](http://www.image-net.org/) and put the downloaded file on \n`dataset/ILSVRC2012_img_train.tar` and `dataset/ILSVRC2012_img_val.tar`.\n\nThen, run the following command on `root` directory to extract the images. \n```\n./dataset/prepare_imagenet.sh\n```\n`apt-get install parallel` may be required to install [parallel](https://www.gnu.org/software/parallel/).\n\nThe structure of image files looks like\n```\ndataset\n└── ILSVRC\n    └── train\n        └── n01440764\n            ├── n01440764_10026.JPEG\n            ├── n01440764_10027.JPEG\n            └── ...\n        └── n01443537\n        └── ...\n    └── val2\n        └── 0\n            ├── 0.jpeg\n            ├── 1.jpeg\n            └── ...\n        └── 1\n        └── ...\n    └── val\n        ├── ILSVRC2012_val_00000001.JPEG\n        ├── ILSVRC2012_val_00000002.JPEG\n        └── ...\n```\nCorresponding annotation files can be found in [here](metadata/ILSVRC).\n\n#### CUB\n\nBoth the original \n[CUB-200-2011](http://www.vision.caltech.edu/visipedia/CUB-200-2011.html) \nand our CUBV2 datasets are required for WSOL training. Note that CUBV2 is \nconsidered as a validation split (`train-fullsup`). Then, run the following \ncommand to download original CUB dataset and extract the image files on `root` \ndirectory.\n\n```\n./dataset/prepare_cub.sh\n```\n\nNote: you can also download the CUBV2 dataset from \n[here](https://drive.google.com/open?id=1U6cwKHS65wayT9FFvoLIA8cn1k0Ot2M1). Put \nthe downloaded file on `dataset/CUBV2.tar` directory and then run the above script.\n\nThe structure of image files looks like\n```\ndataset\n└── CUB\n    └── 001.Black_footed_Albatross\n        ├── Black_Footed_Albatross_0001_796111.jpg\n        ├── Black_Footed_Albatross_0002_55.jpg\n        └── ...\n    └── 002.Laysan_Albatross\n    └── ...\n```\nCorresponding annotation files can be found in [here](metadata/CUB).\n\n\n#### OpenImages\n\nWe provide a new WSOL benchmark, OpenImages30k, \nbased on [OpenImagesV5](https://storage.googleapis.com/openimages/web/index.html). \n\nTo download and extract files, run the following command on `root` directory\n\n```\n./dataset/prepare_openimages.sh\n```\n\nNote: you can also download the OpenImages30k dataset from here \n([images](https://drive.google.com/open?id=1oOb4WQ-lb8SYppHEg3lWnpHk1X8WVO4e)\n, [masks](https://drive.google.com/open?id=1eu1YvcZlsEalhXTS_5Ni5tkImCliIPie)).\nPut the downloaded `OpenImages_images.zip` and `OpenImages_annotations.zip` \nfiles in `dataset` directory and run the above script.\n\nThe structure of image files looks like:\n```\ndataset\n└── OpenImages\n    └── train\n        └── 0bt_c3\n            ├── 0a9b7df4d832baf7.jpg\n            ├── 0abee225b2418fe7.jpg\n            └── ...\n        └── 0bt9lr\n        └── ...\n    └── val\n        └── 0bt_c3\n            ├── 1cd9ac0169ec7df0.jpg\n            ├── 1cd9ac0169ec7df0_ignore.png\n            ├── 1cd9ac0169ec7df0_m0bt_c3_6932e993.png\n            └── ...\n        └── 0bt9lr\n        └── ...\n    └── test   \n        └── 0bt_c3\n            ├── 0a51958fcd523ae4.jpg\n            ├── 0a51958fcd523ae4_ignore.png\n            ├── 0a51958fcd523ae4_m0bt_c3_41344f12.png\n            ├── 0a51958fcd523ae4_m0bt_c3_48f37c0f.png\n            └── ...\n        └── 0bt9lr\n        └── ...\n```\n\nCorresponding annotation files can be found in [here](metadata/OpenImages). \n\n### Run train+eval\n\nWe support the following architecture and method combinations:\n\n* Architectures.\n  - `vgg16`\n  - `inception_v3`\n  - `resnet50`\n  \n* Methods (see [Library of WSOL methods](#5-library-of-wsol-methods) and paper for descriptions).\n  - `cam`\n  - `has`\n  - `acol`\n  - `spg`\n  - `adl`\n  - `cutmix`\n\nBelow is an example command line for the train+eval script.\n```bash\npython main.py --dataset_name OpenImages \\\n               --architecture vgg16 \\\n               --wsol_method cam \\\n               --experiment_name OpenImages_vgg16_CAM \\\n               --pretrained TRUE \\\n               --num_val_sample_per_class 5 \\\n               --large_feature_map FALSE \\\n               --batch_size 32 \\\n               --epochs 10 \\\n               --lr 0.00227913316 \\\n               --lr_decay_frequency 3 \\\n               --weight_decay 5.00E-04 \\\n               --override_cache FALSE \\\n               --workers 4 \\\n               --box_v2_metric True \\\n               --iou_threshold_list 30 50 70 \\\n               --eval_checkpoint_type last\n```\n\nSee [config.py](config.py) for the full descriptions of the arguments, especially \nthe method-specific hyperparameters.\n\nDuring training, we evaluate the model on `train-fullsup` split at every epoch and \nsave a checkpoint (`best_checkpoint.pth.tar`) if the localization performance \nsurpasses every previous score. We also save last checkpoint \n(`last_checkpoint.pth.tar`) when the training is finished. You can select \ncheckpoint type for evaluation on `test` split by setting `eval_checkpoint_type` \nargument accordingly. We suggest to use the last checkpoint for evaluation.\n\n### Improved box evaluation\n\nWe introduce an improved box evaluation metric, `MaxBoxAccV2`, over the original metric used in the CVPR version: `MaxBoxAcc`. \nKey improvements are as follows:\n\n+ Box evaluation using multiple IoU thresholds (default: 30%, 50%, 70%). \nIf you set `multi_iou_eval` to `True` (default), the `localization` metric in the \nlog shows a mean of `MaxBoxAcc` across all IoU thresholds. Otherwise, it only shows \n`MaxBoxAcc` at 50% IoU threshold. The IoU threshold list can be easily set by \nchanging `iou_threshold_list` argument. \n\n+ A new advanced bounding box mining scheme. Bounding boxes are extracted from \nall contours in the thresholded score map. You can use this feature by setting \n`multi_contour_eval` to `True` (default). Otherwise, bounding boxes are extracted \nfrom the largest connected component of the score map.\n\n**We recommend for future researchers to use the `MaxBoxAccV2` metric for box-based evaluation.**\nUsers can evaluate the WSOL methods with this metric by setting `box_v2_metric` to `True`.\n\n## 7. Code license\n\nThis project is distributed under MIT license.\n\n```\nCopyright (c) 2020-present NAVER Corp.\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n```\n\n## 8. How to cite\n\n```\n@inproceedings{choe2020cvpr,\n  title={Evaluating Weakly Supervised Object Localization Methods Right},\n  author={Choe, Junsuk and Oh, Seong Joon and Lee, Seungho and Chun, Sanghyuk and Akata, Zeynep and Shim, Hyunjung},\n  year = {2020},\n  booktitle = {Conference on Computer Vision and Pattern Recognition (CVPR)},\n  note = {to appear},\n  pubstate = {published},\n  tppubtype = {inproceedings}\n}\n```\n```\n@ARTICLE{choe2022tpami,\n  author={Choe, Junsuk and Oh, Seong Joon and Chun, Sanghyuk and Lee, Seungho and Akata, Zeynep and Shim, Hyunjung},\n  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, \n  title={Evaluation for Weakly Supervised Object Localization: Protocol, Metrics, and Datasets}, \n  year={2022},\n  volume={},\n  number={},\n  pages={1-1},\n  doi={10.1109/TPAMI.2022.3169881}}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclovaai%2Fwsolevaluation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fclovaai%2Fwsolevaluation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclovaai%2Fwsolevaluation/lists"}