{"id":32385436,"url":"https://github.com/larsvanderlaan/selfcalibratingconformal","last_synced_at":"2025-10-25T02:51:56.614Z","repository":{"id":218961994,"uuid":"747807959","full_name":"Larsvanderlaan/SelfCalibratingConformal","owner":"Larsvanderlaan","description":"An implementation of Self-Calibrating Conformal Prediction, accepted to Neurips 2024. SC-CP combines Venn-Abers calibration and conformal prediction to deliver calibrated point predictions alongside prediction intervals with finite-sample validity conditional on these predictions.","archived":false,"fork":false,"pushed_at":"2025-01-28T16:09:47.000Z","size":84040,"stargazers_count":8,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-08-30T20:16:47.327Z","etag":null,"topics":["calibration","conformal-prediction","isotonic-calibration","prediction-intervals","predictive-inference","uncertainty-quantification","venn-abers"],"latest_commit_sha":null,"homepage":"https://openreview.net/pdf?id=BJ6HkT7qIk","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Larsvanderlaan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-24T17:20:24.000Z","updated_at":"2025-06-18T16:57:46.000Z","dependencies_parsed_at":"2024-05-16T05:12:24.667Z","dependency_job_id":null,"html_url":"https://github.com/Larsvanderlaan/SelfCalibratingConformal","commit_stats":null,"previous_names":["larsvanderlaan/conformal","larsvanderlaan/selfcalibratingconformal"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Larsvanderlaan/SelfCalibratingConformal","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Larsvanderlaan%2FSelfCalibratingConformal","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Larsvanderlaan%2FSelfCalibratingConformal/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Larsvanderlaan%2FSelfCalibratingConformal/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Larsvanderlaan%2FSelfCalibratingConformal/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Larsvanderlaan","download_url":"https://codeload.github.com/Larsvanderlaan/SelfCalibratingConformal/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Larsvanderlaan%2FSelfCalibratingConformal/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":280897698,"owners_count":26409960,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-25T02:00:06.499Z","response_time":81,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["calibration","conformal-prediction","isotonic-calibration","prediction-intervals","predictive-inference","uncertainty-quantification","venn-abers"],"created_at":"2025-10-25T02:51:23.015Z","updated_at":"2025-10-25T02:51:56.606Z","avatar_url":"https://github.com/Larsvanderlaan.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Self-Calibrating Conformal Prediction: Providing Model Calibration and Predictive Inference\n\nThis repository provides a Python implementation of **Self-Calibrating Conformal Prediction** (previously termed *Self-Consistent Conformal Prediction*), associated with the [NeurIPS 2024 conference paper](https://arxiv.org/abs/2402.07307). It also contains code to reproduce the experiments from the paper.\n\n\n## Installation\n\nRun the following command to install the package:\n\n```bash\npip install SelfCalibratingConformal\n```\n\n## Abstract\n\nIn machine learning, model calibration and predictive inference are essential for producing reliable predictions and quantifying uncertainty to support decision-making. Recognizing the complementary roles of point and interval predictions, we introduce Self-Calibrating Conformal Prediction, a method that combines Venn-Abers calibration and conformal prediction to deliver calibrated point predictions alongside prediction intervals with finite-sample validity conditional on these predictions. To achieve this, we extend the original Venn-Abers procedure from binary classification to regression. Our theoretical framework supports analyzing conformal prediction methods that involve calibrating model predictions and subsequently constructing conditionally valid prediction intervals on the same data, where the conditioning set or conformity scores may depend on the calibrated predictions. Real-data experiments show that our method improves interval efficiency through model calibration and offers a practical alternative to feature-conditional validity.\n\n## Example\n\nExample code demonstrating the use of **Self-Calibrating Conformal Prediction** can be found in the `vignette.ipynb` file. This notebook includes code to perform model calibration, generate prediction intervals, and evaluate model coverage.\n\n```{python}\n# See vignette.ipynb for loading data\n\n# Fit an XGBoost model (or any predictive model)\nparams = {\n    'max_depth': 4,\n    'learning_rate': 0.05,\n    'n_estimators': 100\n}\nmodel = xgb.XGBRegressor(**params)\nmodel.fit(X_train, y_train)\n\n# Define a predictor function using the fitted model\ndef predictor(x):\n    return model.predict(np.array(x))\n\n# Set significance level for conformal prediction\nalpha = 0.1\n\n# Apply Self-Calibrating Conformal Prediction\nconformal_predictor = SelfCalibratingConformalPredictor(predictor, algo_params={'num_bin_predictor': 200, 'num_bin_y': 100, 'binning_method': \"quantile\"})\nconformal_predictor.calibrate(X_cal, y_cal, alpha=alpha)\n\n# Calibrated point predictions derived from Venn-Abers\nprediction_calibrated = conformal_predictor.predict_point(X_test)\n\n# Worst and best case bounds for Venn-Abers calibrated multi-prediction\nprediction_venn_abers = conformal_predictor.predict_venn_abers(X_test)\n\n# Self-calibrated prediction interval\nprediction_interval = conformal_predictor.predict_interval(X_test)\n\n# Evaluate coverage and average interval width on the test set\ncoverage, width = conformal_predictor.check_coverage(X_test, y_test)\n```\n\n\n## Citation\nvan der Laan, L., \u0026 Alaa, A. M. (2024). Self-Calibrating Conformal Prediction. arXiv preprint arXiv:2402.07307.\n\n\n\n@misc{vanderlaan2024selfcalibratingconformalprediction,\n      title={Self-Calibrating Conformal Prediction}, \n      author={Lars van der Laan and Ahmed M. Alaa},\n      year={2024},\n      eprint={2402.07307},\n      archivePrefix={arXiv},\n      primaryClass={stat.ML},\n      url={https://arxiv.org/abs/2402.07307}, \n}\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flarsvanderlaan%2Fselfcalibratingconformal","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flarsvanderlaan%2Fselfcalibratingconformal","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flarsvanderlaan%2Fselfcalibratingconformal/lists"}