https://github.com/larsvanderlaan/selfcalibratingconformal
An implementation of Self-Calibrating Conformal Prediction, accepted to Neurips 2024. SC-CP combines Venn-Abers calibration and conformal prediction to deliver calibrated point predictions alongside prediction intervals with finite-sample validity conditional on these predictions.
https://github.com/larsvanderlaan/selfcalibratingconformal
calibration conformal-prediction isotonic-calibration prediction-intervals predictive-inference uncertainty-quantification venn-abers
Last synced: 8 months ago
JSON representation
An implementation of Self-Calibrating Conformal Prediction, accepted to Neurips 2024. SC-CP combines Venn-Abers calibration and conformal prediction to deliver calibrated point predictions alongside prediction intervals with finite-sample validity conditional on these predictions.
- Host: GitHub
- URL: https://github.com/larsvanderlaan/selfcalibratingconformal
- Owner: Larsvanderlaan
- Created: 2024-01-24T17:20:24.000Z (over 2 years ago)
- Default Branch: master
- Last Pushed: 2025-01-28T16:09:47.000Z (over 1 year ago)
- Last Synced: 2025-08-30T20:16:47.327Z (10 months ago)
- Topics: calibration, conformal-prediction, isotonic-calibration, prediction-intervals, predictive-inference, uncertainty-quantification, venn-abers
- Language: Jupyter Notebook
- Homepage: https://openreview.net/pdf?id=BJ6HkT7qIk
- Size: 80.1 MB
- Stars: 8
- Watchers: 1
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Self-Calibrating Conformal Prediction: Providing Model Calibration and Predictive Inference
This repository provides a Python implementation of **Self-Calibrating Conformal Prediction** (previously termed *Self-Consistent Conformal Prediction*), associated with the [NeurIPS 2024 conference paper](https://arxiv.org/abs/2402.07307). It also contains code to reproduce the experiments from the paper.
## Installation
Run the following command to install the package:
```bash
pip install SelfCalibratingConformal
```
## Abstract
In machine learning, model calibration and predictive inference are essential for producing reliable predictions and quantifying uncertainty to support decision-making. Recognizing the complementary roles of point and interval predictions, we introduce Self-Calibrating Conformal Prediction, a method that combines Venn-Abers calibration and conformal prediction to deliver calibrated point predictions alongside prediction intervals with finite-sample validity conditional on these predictions. To achieve this, we extend the original Venn-Abers procedure from binary classification to regression. Our theoretical framework supports analyzing conformal prediction methods that involve calibrating model predictions and subsequently constructing conditionally valid prediction intervals on the same data, where the conditioning set or conformity scores may depend on the calibrated predictions. Real-data experiments show that our method improves interval efficiency through model calibration and offers a practical alternative to feature-conditional validity.
## Example
Example code demonstrating the use of **Self-Calibrating Conformal Prediction** can be found in the `vignette.ipynb` file. This notebook includes code to perform model calibration, generate prediction intervals, and evaluate model coverage.
```{python}
# See vignette.ipynb for loading data
# Fit an XGBoost model (or any predictive model)
params = {
'max_depth': 4,
'learning_rate': 0.05,
'n_estimators': 100
}
model = xgb.XGBRegressor(**params)
model.fit(X_train, y_train)
# Define a predictor function using the fitted model
def predictor(x):
return model.predict(np.array(x))
# Set significance level for conformal prediction
alpha = 0.1
# Apply Self-Calibrating Conformal Prediction
conformal_predictor = SelfCalibratingConformalPredictor(predictor, algo_params={'num_bin_predictor': 200, 'num_bin_y': 100, 'binning_method': "quantile"})
conformal_predictor.calibrate(X_cal, y_cal, alpha=alpha)
# Calibrated point predictions derived from Venn-Abers
prediction_calibrated = conformal_predictor.predict_point(X_test)
# Worst and best case bounds for Venn-Abers calibrated multi-prediction
prediction_venn_abers = conformal_predictor.predict_venn_abers(X_test)
# Self-calibrated prediction interval
prediction_interval = conformal_predictor.predict_interval(X_test)
# Evaluate coverage and average interval width on the test set
coverage, width = conformal_predictor.check_coverage(X_test, y_test)
```
## Citation
van der Laan, L., & Alaa, A. M. (2024). Self-Calibrating Conformal Prediction. arXiv preprint arXiv:2402.07307.
@misc{vanderlaan2024selfcalibratingconformalprediction,
title={Self-Calibrating Conformal Prediction},
author={Lars van der Laan and Ahmed M. Alaa},
year={2024},
eprint={2402.07307},
archivePrefix={arXiv},
primaryClass={stat.ML},
url={https://arxiv.org/abs/2402.07307},
}