{"id":16863546,"url":"https://github.com/jonathanwenger/pycalib","last_synced_at":"2025-04-11T08:58:50.819Z","repository":{"id":115615541,"uuid":"183919280","full_name":"JonathanWenger/pycalib","owner":"JonathanWenger","description":"Non-Parametric Calibration for Classification (AISTATS 2020)","archived":false,"fork":false,"pushed_at":"2022-02-03T22:02:29.000Z","size":208202,"stargazers_count":19,"open_issues_count":0,"forks_count":9,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-25T06:24:19.885Z","etag":null,"topics":["aistats-2020","machine-learning","neural-networks","probability-calibration"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JonathanWenger.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-04-28T14:32:33.000Z","updated_at":"2024-11-18T09:41:18.000Z","dependencies_parsed_at":null,"dependency_job_id":"587e871a-6e5f-434b-b165-f0ebf74d7638","html_url":"https://github.com/JonathanWenger/pycalib","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JonathanWenger%2Fpycalib","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JonathanWenger%2Fpycalib/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JonathanWenger%2Fpycalib/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JonathanWenger%2Fpycalib/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JonathanWenger","download_url":"https://codeload.github.com/JonathanWenger/pycalib/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248363631,"owners_count":21091405,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aistats-2020","machine-learning","neural-networks","probability-calibration"],"created_at":"2024-10-13T14:38:59.008Z","updated_at":"2025-04-11T08:58:50.811Z","avatar_url":"https://github.com/JonathanWenger.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Non-Parametric Calibration for Classification\n\n[![arXiv](https://img.shields.io/static/v1?logo=arxiv\u0026logoColor=white\u0026label=Preprint\u0026message=1906.04933\u0026color=B31B1B)](https://arxiv.org/abs/1906.04933)\n[![Build Status](https://travis-ci.org/JonathanWenger/pycalib.svg?branch=master)](https://travis-ci.org/JonathanWenger/pycalib)\n\nThis repository provides an implementation of the paper [\"Non-Parametric Calibration for Classification\"](https://arxiv.org/abs/1906.04933) (Jonathan Wenger, Hedvig Kjellström, Rudolph Triebel) published at AISTATS 2020. All results presented in our work were produced with this code.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/JonathanWenger/pycalib/master/figures/ece_logloss_example/ece_logloss.png\" alt=\"latent_process\" width=\"700\"/\u003e\n\u003c/p\u003e\n\n\n## Introduction\n\nMany applications of classification methods not only require high accuracy but also reliable estimation of predictive uncertainty. However, while many current classification frameworks, in particular deep neural networks, achieve high accuracy, they tend to incorrectly estimate uncertainty. We provide a method that adjusts the confidence estimates of a general classifier such that they approach the probability of classifying correctly. In contrast to existing approaches, our calibration method employs a non-parametric representation using a latent Gaussian process, and is specifically designed for multi-class classification. It can be applied to any classifier that outputs confidence estimates and is not limited to neural networks. In the experiments included in this repository, we show the universally strong performance of our method across different classifiers and benchmark data sets, in particular for state-of-the art neural network architectures.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/JonathanWenger/pycalib/master/figures/gpcalib_illustration/diagram_calibration.png\" alt=\"diagram_calibration\" width=\"400\"/\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/JonathanWenger/pycalib/master/figures/gpcalib_illustration/latent_process.png\" alt=\"latent_process\" width=\"400\"/\u003e\n\u003c/p\u003e\n\n## Installation and Documentation\nThe code was developed in Python 3.6 under Ubuntu (18.04). You can install this Python 3 package using `pip` (or `pip3`):\n```bash\npip install setuptools numpy scipy scikit-learn cython\npip install git+https://github.com/JonathanWenger/pycalib.git\n```\nNote that some dependencies need to be installed separately since a subset of experiments rely on \n[scikit-garden](https://github.com/scikit-garden/scikit-garden). You can also clone this repository to run the scripts\n reproducing the experiments in the paper via:\n```bash\npip install setuptools numpy scipy scikit-learn cython\npip install -e git+git://github.com/scikit-garden/scikit-garden.git#egg=scikit-garden\ngit clone https://github.com/JonathanWenger/pycalib\ncd pycalib\npython setup.py install\n```\nFor tips on getting started and how to use this package please refer to the [documentation](https://jonathanwenger.github.io/pycalib/).\n\n## Experiments and Benchmark Datasets\n\nWe performed calibration experiments for binary and multi-class benchmark datasets from computer vision for a range of classifiers and calibration methods. We found that GP calibration performed particularly well on large-scale architectures and challenging data sets.\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/JonathanWenger/pycalib/master/figures/gpcalib_illustration/calibration_experiment_results.png\" alt=\"latent_process\" width=\"700\"/\u003e\n\u003c/p\u003e\n\nThe experiments can be reproduced by using the scripts in `benchmark` and `figures`. The datasets we used and how to obtain them are listed below.\n\n- **PCam**: Due to the size of the data, only a script replicating the experiments is provided. The data can be downloaded from the\n[PCam repository](https://github.com/basveeling/pcam).\n- **KITTI**: The repository includes 64-dimensional features extracted from KITTI sequences compressed in a zip file\n`datasets/kitti/kitti_data.zip`.\n- **MNIST**: A script will automatically download the MNIST dataset if needed.\n- **CIFAR-100**: When the CIFAR-100 experiment is run, there is an option to automatically download the dataset.\n- **ImageNet 2012**: Due to the size of the data, only a script replicating the experiments is provided. The ImageNet validation data can be\nobtained from the [ImageNet website](http://www.image-net.org).\n\n\n## Publication\nIf you use this repository in your research, please cite the following paper:\n\nJ. Wenger, H. Kjellström, and R. Triebel. Non-parametric calibration for classification ([PDF](https://arxiv.org/pdf/1906.04933.pdf)). In _Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS)_.\n\n\n```bibtex\n@InProceedings{wenger2020calibration,\n  title         = {Non-Parametric Calibration for Classification},\n  author        = {Jonathan Wenger and Hedvig Kjellstr{\\\"o}m and Rudolph Triebel},\n  booktitle     = {Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS)},\n  year          = {2020},\n  series        = {Proceedings of Machine Learning Research},\n  keywords      = {calibration, non-parametric, gaussian processes, classification},\n  url           = {https://github.com/JonathanWenger/pycalib}\n}\n```\n\n## License and Contact\n\nThis work is released under the [MIT License](https://github.com/JonathanWenger/pycalib/blob/master/LICENSE).\n\nPlease submit an [issue on GitHub](https://github.com/JonathanWenger/pycalib/issues/new) to report bugs or request changes.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonathanwenger%2Fpycalib","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjonathanwenger%2Fpycalib","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonathanwenger%2Fpycalib/lists"}