{"id":13577577,"url":"https://github.com/Alibaba-MIIL/PartialLabelingCSL","last_synced_at":"2025-04-05T12:30:47.642Z","repository":{"id":40375387,"uuid":"417113517","full_name":"Alibaba-MIIL/PartialLabelingCSL","owner":"Alibaba-MIIL","description":"Official implementation for the paper: \"Multi-label Classification with Partial Annotations using Class-aware Selective Loss\"","archived":false,"fork":false,"pushed_at":"2022-08-23T15:21:33.000Z","size":4871,"stargazers_count":129,"open_issues_count":10,"forks_count":18,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-11-05T14:46:22.383Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Alibaba-MIIL.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-10-14T12:02:59.000Z","updated_at":"2024-09-10T12:38:09.000Z","dependencies_parsed_at":"2022-09-07T09:51:26.578Z","dependency_job_id":null,"html_url":"https://github.com/Alibaba-MIIL/PartialLabelingCSL","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alibaba-MIIL%2FPartialLabelingCSL","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alibaba-MIIL%2FPartialLabelingCSL/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alibaba-MIIL%2FPartialLabelingCSL/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alibaba-MIIL%2FPartialLabelingCSL/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Alibaba-MIIL","download_url":"https://codeload.github.com/Alibaba-MIIL/PartialLabelingCSL/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247338546,"owners_count":20922986,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T15:01:22.650Z","updated_at":"2025-04-05T12:30:42.635Z","avatar_url":"https://github.com/Alibaba-MIIL.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/multi-label-classification-with-partial/multi-label-classification-on-openimages-v6)](https://paperswithcode.com/sota/multi-label-classification-on-openimages-v6?p=multi-label-classification-with-partial)\n\n# Multi-label Classification with Partial Annotations using Class-aware Selective Loss\n\n\u003cbr\u003e [Paper](https://arxiv.org/abs/2110.10955) |\n[Pretrained models](https://github.com/Alibaba-MIIL/PartialLabelingCSL/blob/main/README.md#pretrained-models) |\n[OpenImages download](https://github.com/Alibaba-MIIL/PartialLabelingCSL/blob/main/OpenImages.md)\n\nOfficial PyTorch Implementation\n\n\u003e Emanuel Ben-Baruch, Tal Ridnik, Itamar Friedman, Avi Ben-Cohen, Nadav Zamir, Asaf Noy, Lihi Zelnik-Manor\u003cbr/\u003e DAMO Academy, Alibaba\n\u003e Group\n\n**Abstract**\n\nLarge-scale multi-label classification datasets are commonly, and perhaps inevitably, partially annotated. That is, only a small subset of labels are annotated per sample.\nDifferent methods for handling the missing labels induce different properties on the model and impact its accuracy.\nIn this work, we analyze the partial labeling problem, then propose a solution based on two key ideas. \nFirst, un-annotated labels should be treated selectively according to two probability quantities: the class distribution in the overall dataset and the specific label likelihood for a given data sample.\nWe propose to estimate the class distribution using a dedicated temporary model, and we show its improved efficiency over a naive estimation computed using the dataset's partial annotations.\nSecond, during the training of the target model, we emphasize the contribution of annotated labels over originally un-annotated labels by using a dedicated asymmetric loss.\nExperiments conducted on three partially labeled datasets, OpenImages, LVIS, and simulated-COCO, demonstrate the effectiveness of our approach. Specifically, with our novel selective approach, we achieve state-of-the-art results on OpenImages dataset. \n\n\u003c!-- ### Challenges in Partial Labeling\n(a) In a partially labeled dataset, only a portion of the samples are annotated for a given class. (b) \"Ignore\" mode exploits only the annotated samples which may lead to a limited decision boundary. (c) \"Negative\" mode naively treats all un-annotated labels as negatives. It may produce suboptimal decision boundary as it adds noise of un-annotated positive labels. Also, annotated and un-annotated negative samples contribute similarly to the optimization. (d) Our approach aims at mitigating these drawbacks by predicting the probability of a label being present in the image.\n \n\u003cp align=\"center\"\u003e\n \u003ctable class=\"tg\"\u003e\n  \u003ctr\u003e\n    \u003ctd class=\"tg-c3ow\"\u003e\u003cimg src=\"./pics/intro_modes_CSL.png\" align=\"center\" width=\"1000\" \u003e\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\u003c/p\u003e --\u003e\n\n### Direct OpenImages Download is Now Available. \nWe provide direct and convenient access for the OpenImages (V6) dataset. This will enable a common and reproducible baseline for benchmarking and future research. See further details [here](https://github.com/Alibaba-MIIL/PartialLabelingCSL/blob/main/OpenImages.md).\n\n### Class-aware Selective Approach\nAn overview of our approach is summarized in the following figure:\n \n\u003cp align=\"center\"\u003e\n \u003ctable class=\"tg\"\u003e\n  \u003ctr\u003e\n    \u003ctd class=\"tg-c3ow\"\u003e\u003cimg src=\"./pics/CSL_approach.png\" align=\"center\" width=\"900\" \u003e\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\u003c/p\u003e\n\n### Loss Implementation \n\nOur loss consists of a selective approach that adjusts the training mode for each class individually and a partial asymmetric loss.\n\u003c!-- The selective approach is based on two probabilities quantities: label likelihood and label prior. The partial asymmetric loss emphasizes the contribution of the annotated labels.   --\u003e\nAn implementation of the Class-aware Selective Loss (CSL) can be found [here](/src/loss_functions/partial_asymmetric_loss.py). \n- ```class PartialSelectiveLoss(nn.Module)```\n\n\n## Pretrained Models\n\u003c!-- In this [link](MODEL_ZOO.md), we provide pre-trained models on various dataset.  --\u003e\nWe provide models pretrained on the OpenImages dataset with different partial training-modes and architectures:\n\n| Model | Architecture | Link | mAP |\n| :---            | :---:      | :---:     | ---: |\n| Ignore          | TResNet-M | [link](https://miil-public-eu.oss-eu-central-1.aliyuncs.com/model-zoo/CSL/opim_v6/mtresnet_opim_ignore.pth)      | 85.38       |\n| Negative        | TResNet-M | [link](https://miil-public-eu.oss-eu-central-1.aliyuncs.com/model-zoo/CSL/opim_v6/mtresnet_opim_negative.pth)    | 85.85       |\n| Selective (CSL) | TResNet-M  | [link](https://miil-public-eu.oss-eu-central-1.aliyuncs.com/model-zoo/CSL/opim_v6/mtresnet_opim_86.72.pth)      | 86.72       |\n| Selective (CSL) | TResNet-L  | [link](https://miil-public-eu.oss-eu-central-1.aliyuncs.com/model-zoo/CSL/opim_v6/ltresnet_v2_opim_87.34.pth)   | **87.34**   |\n \n\n\n## Inference Code (Demo)\nWe provide [inference code](infer.py), that demonstrates how to load the\nmodel, pre-process an image and do inference. An example run of\nOpenImages model (after downloading the relevant model):\n```\npython infer.py  \\\n--dataset_type=OpenImages \\\n--model_name=tresnet_m \\\n--model_path=./models_local/mtresnet_opim_86.72.pth \\\n--pic_path=./pics/10162266293_c7634cbda9_o.jpg \\\n--input_size=224\n```\n\n### Result Examples \n\u003cp align=\"center\"\u003e\n \u003ctable class=\"tg\"\u003e\n  \u003ctr\u003e\n    \u003ctd class=\"tg-c3ow\"\u003e\u003cimg src=\"./pics/demo_examples.png\" align=\"center\" width=\"900\" \u003e\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\u003c/p\u003e\n\n\n\n## Training Code\nTraining code is provided in [train.py](train.py). Also, code for simulating partial annotation for the [MS-COCO dataset](https://cocodataset.org/#download) is available ([coco_simulation](src/helper_functions/coco_simulation.py)). In particular, two \"partial\" simulation schemes are implemented: fix-per-class(FPC) and random-per-sample (RPS).\n- FPC: For each class, we randomly sample a fixed number of positive annotations and the same number of negative annotations.  The rest of the annotations are dropped.\n- RPS: We omit each annotation with probability p.\n\nPretrained weights using the ImageNet-21k dataset can be found here: [link](https://github.com/Alibaba-MIIL/ImageNet21K/blob/main/MODEL_ZOO.md)\\\nPretrained weights using the ImageNet-1k dataset can be found here: [link](https://github.com/Alibaba-MIIL/TResNet/blob/master/MODEL_ZOO.md)\n\nExample of training with RPS simulation:\n```\n--data=/datasets/COCO/COCO_2014\n--model-path=models/pretrain/mtresnet_21k\n--gamma_pos=0\n--gamma_neg=1\n--gamma_unann=4\n--simulate_partial_type=rps\n--simulate_partial_param=0.5\n--partial_loss_mode=selective\n--likelihood_topk=5\n--prior_threshold=0.5\n--prior_path=./outputs/priors/prior_fpc_1000.csv\n```\n\nExample of training with FPC simulation:\n```\n--data=/mnt/datasets/COCO/COCO_2014\n--model-path=models/pretrain/mtresnet_21k\n--gamma_pos=0\n--gamma_neg=3\n--gamma_unann=4\n--simulate_partial_type=fpc\n--simulate_partial_param=1000\n--partial_loss_mode=selective\n--likelihood_topk=5\n--prior_threshold=0.5\n--prior_path=./outputs/priors/prior_fpc_1000.csv\n```\n\n### Typical Training Results\n\n#### FPC (1,000) simulation scheme:\n| Model                    | mAP         | \n| :---                     | :---:       |\n| Ignore, CE               | 76.46       |\n| Negative, CE             | 81.24       |\n| Negative, ASL (4,1)      | 81.64       |\n| CSL - Selective, P-ASL(4,3,1)  | **83.44**       |     \n\n\n#### RPS (0.5) simulation scheme:\n| Model                    | mAP        | \n| :---                     | :---:      |\n| Ignore, CE               | 84.90      |\n| Negative, CE             | 81.21      |\n| Negative, ASL (4,1)      | 81.91      |\n| CSL- Selective, P-ASL(4,1,1)  | **85.21**      |  \n\n\n## Estimating the Class Distribution\nThe training code contains also the procedure for estimating the class distribution from the data. Our approach enables us to rank the classes based on predictions of a temporary model trained using the *Ignore* mode. [link](https://github.com/Alibaba-MIIL/PartialLabelingCSL/blob/cadc2afab73294a0e9e0799eec06b095e50e646e/src/loss_functions/partial_asymmetric_loss.py#L131)\n\n#### Top 10 classes:\n| Method                    | Top 10 ranked classes      | \n| :---                     | :---:      |\n| Original                           | 'person', 'chair', 'car', 'dining table', 'cup', 'bottle', 'bowl', 'handbag', 'truck', 'backpack'   |\n| Estiimate (Ignore mode)            | 'person', 'chair', 'handbag', 'cup', 'bench', 'bottle', 'backpack', 'car', 'cell phone', 'potted plant'      |\n| Estimate (Negative mode)           | 'kite' 'truck' 'carrot' 'baseball glove' 'tennis racket' 'remote' 'cat' 'tie' 'horse' 'boat'    |\n\n\n## Citation\n```\n@misc{benbaruch2021multilabel,\n      title={Multi-label Classification with Partial Annotations using Class-aware Selective Loss}, \n      author={Emanuel Ben-Baruch and Tal Ridnik and Itamar Friedman and Avi Ben-Cohen and Nadav Zamir and Asaf Noy and Lihi Zelnik-Manor},\n      year={2021},\n      eprint={2110.10955},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV}\n}\n```\n\n## Acknowledgements\nSeveral images from [OpenImages dataset](https://storage.googleapis.com/openimages/web/index.html) are used in this project.\nSome components of this code implementation are adapted from the repository https://github.com/Alibaba-MIIL/ASL.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAlibaba-MIIL%2FPartialLabelingCSL","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FAlibaba-MIIL%2FPartialLabelingCSL","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAlibaba-MIIL%2FPartialLabelingCSL/lists"}