https://github.com/msesia/conformal-label-noise

This repository implements conformal prediction methods for classification tasks that can automatically adapt to random label contamination in the calibration sample.
https://github.com/msesia/conformal-label-noise

classification computer-vision conformal-prediction label-noise uncertainty-quantification

Last synced: about 2 months ago
JSON representation

This repository implements conformal prediction methods for classification tasks that can automatically adapt to random label contamination in the calibration sample.

• Host: GitHub
• URL: https://github.com/msesia/conformal-label-noise
• Owner: msesia
• License: mit
• Created: 2023-09-10T16:52:04.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2024-06-19T05:42:27.000Z (3 months ago)
• Last Synced: 2024-07-23T05:18:44.210Z (2 months ago)
• Topics: classification, computer-vision, conformal-prediction, label-noise, uncertainty-quantification
• Language: Python
• Homepage:
• Size: 133 KB
• Stars: 2
• Watchers: 2
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Adaptive Conformal Classification with Noisy Labels

This repository provides the software implementation of methods from the paper:

>  "Adaptive Conformal Classification with Noisy Labels"  

>  Matteo Sesia, Y. X. Rachel Wang, Xin Tong  

>  [arXiv preprint](https://arxiv.org/abs/2309.05092)

## Abstract

We present a conformal prediction method for classification tasks that can adapt to random label contamination in the calibration sample, often leading to more informative prediction sets with stronger coverage guarantees compared to existing approaches. This is obtained through a precise characterization of the coverage inflation (or deflation) suffered by standard conformal inferences in the presence of label contamination, which is then made actionable through a new calibration algorithm. Our solution can leverage different modeling assumptions about the contamination process, while requiring no knowledge of the underlying data distribution or of the inner workings of the classification model. The performance of the proposed method is demonstrated through simulations and an application to object classification with the CIFAR-10H image data set.

## Contents

- `cln/`: Python package implementing the methods from the paper.

- `third_party/`: Third-party Python packages used by this package.

- `examples/`: Jupyter notebooks with introductory examples.

- `experiments/`: Code for reproducing the numerical experiments with simulated and real data.

## Prerequisites

For the `cln` package:

- `numpy` (>= 1.25.0)

- `scipy` (>= 1.11.1)

- `scikit-learn` (>= 1.3.0)

- `pandas` (>= 2.0.3)

- `torch` (>= 1.10.2)

- `tqdm` (>= 4.65.0)

- `statsmodels` (>= 0.14.0)

For numerical experiments involving CIFAR-10H data:

- [PyTorch CIFAR-10](https://github.com/huyvnphan/PyTorch_CIFAR10) (included in the 'third_party/' directory)

## Installation

Clone the development version from GitHub:

    git clone https://github.com/msesia/conformal-label-noise.git

## Reproducibility Instructions

See the [experiments/README.md](experiments/README.md) file for instructions on how to obtain the real CIFAR-10 data used in the manuscript and for reproducing the paper's figures.

## License

This project is licensed under the MIT License. See the [LICENSE](LICENSE.txt) file for details.