Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/tiffanyding/class-conditional-conformal
https://github.com/tiffanyding/class-conditional-conformal
Last synced: 9 days ago
JSON representation
- Host: GitHub
- URL: https://github.com/tiffanyding/class-conditional-conformal
- Owner: tiffanyding
- Created: 2023-01-27T19:07:13.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-02-25T10:33:34.000Z (9 months ago)
- Last Synced: 2024-08-02T15:12:06.894Z (3 months ago)
- Language: Jupyter Notebook
- Size: 25.8 MB
- Stars: 12
- Watchers: 1
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
This is the code release accompanying the paper [Class-Conditional Conformal Prediction with Many Classes](https://arxiv.org/abs/2306.09335)
Citation:
```
@article{ding2023classconditional,
title={Class-Conditional Conformal Prediction with Many Classes},
author={Ding, Tiffany and Angelopoulos, Anastasios N and Bates,
Stephen and Jordan, Michael I and Tibshirani, Ryan J},
journal={arXiv preprint arXiv:2306.09335},
year={2023}
}
```## Setup
First, create a virtual environment and install the necessary packages by running
```
conda create --name env
conda activate env
pip install -r requirements.txt
```To make the environment accessible from Jupyter notebooks, run
```
ipython3 kernel install --user --name=conformal_env
```This adds a kernel called `conformal_env` to your list of Jupyter kernels.
Download the datasets by running
```
sh download_data.sh
```which will create a folder called `data/` and download the data described in the following section.
## Data description
1. `imagenet` (4.62 GB): `(115301, 1000)` array of softmax scores and `(115301,)` array of labels
1. `cifar-100` (0.01 GB): `(30000, 100)` array of softmax scores and `(30000,)` array of labels
1. `places365` (0.54 GB): `(183996, 365)` array of softmax scores and `(183996,)` array of labels
1. `inaturalist` (6.72 GB): `(1324900, 633)` array of softmax scores and `(1324900,)` array of labelsThe code for training models on the raw datasets to produce the softmax scores is located in `generate_scores/`
## Running Clustered Conformal
See `example.ipynb` for an example of how to run clustered conformal prediction.
## Reproducing our experiments
Run `sh run_experiments.sh` to run our main set of experiments. Run `sh run_heatmap_experiments.sh` for experiments that test the sensitivity of clustered conformal to the hyperparameter values. To view the main results, run `jupyter notebook` from Terminal, then run the notebooks in the `notebooks/` directory.