{"id":13487586,"url":"https://github.com/uncertainty-toolbox/uncertainty-toolbox","last_synced_at":"2025-05-15T13:08:54.284Z","repository":{"id":39878555,"uuid":"293174301","full_name":"uncertainty-toolbox/uncertainty-toolbox","owner":"uncertainty-toolbox","description":"Uncertainty Toolbox: a Python toolbox for predictive uncertainty quantification, calibration, metrics, and visualization","archived":false,"fork":false,"pushed_at":"2025-03-05T00:46:24.000Z","size":4309,"stargazers_count":1877,"open_issues_count":14,"forks_count":136,"subscribers_count":35,"default_branch":"main","last_synced_at":"2025-04-11T22:36:48.624Z","etag":null,"topics":["bayesian-deep-learning","bayesian-neural-networks","calibration","metrics","predictive-uncertainty","recalibration","scoring-rules","sharpness","toolbox","uncertainty","uncertainty-calibration","uncertainty-estimation","uncertainty-metrics","uncertainty-quantification","uncertainty-toolbox","visualization","visualizations"],"latest_commit_sha":null,"homepage":"https://uncertainty-toolbox.github.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/uncertainty-toolbox.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-09-06T00:12:39.000Z","updated_at":"2025-04-11T08:25:07.000Z","dependencies_parsed_at":"2024-11-20T07:06:33.447Z","dependency_job_id":"150f5adc-d2a2-401f-8c8d-30f7d4a354f1","html_url":"https://github.com/uncertainty-toolbox/uncertainty-toolbox","commit_stats":{"total_commits":131,"total_committers":7,"mean_commits":"18.714285714285715","dds":0.4732824427480916,"last_synced_commit":"6ea1fed6591923a95d49d8049a197e33d4d8092d"},"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uncertainty-toolbox%2Funcertainty-toolbox","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uncertainty-toolbox%2Funcertainty-toolbox/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uncertainty-toolbox%2Funcertainty-toolbox/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uncertainty-toolbox%2Funcertainty-toolbox/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/uncertainty-toolbox","download_url":"https://codeload.github.com/uncertainty-toolbox/uncertainty-toolbox/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254346625,"owners_count":22055808,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bayesian-deep-learning","bayesian-neural-networks","calibration","metrics","predictive-uncertainty","recalibration","scoring-rules","sharpness","toolbox","uncertainty","uncertainty-calibration","uncertainty-estimation","uncertainty-metrics","uncertainty-quantification","uncertainty-toolbox","visualization","visualizations"],"created_at":"2024-07-31T18:01:00.899Z","updated_at":"2025-05-15T13:08:49.275Z","avatar_url":"https://github.com/uncertainty-toolbox.png","language":"Python","funding_links":[],"categories":["Other Toolboxes","Python","\u003cspan id=\"head30\"\u003e3.4. Bayesian Inference\u003c/span\u003e"],"sub_categories":["\u003cspan id=\"head38\"\u003e3.4.8. Uncertanty\u003c/span\u003e"],"readme":"\u003cp align=\"center\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/uncertainty-toolbox/uncertainty-toolbox/master/docs/images/logo.svg\" width=730 /\u003e\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://uncertainty-toolbox.github.io/\"\u003eWebsite\u003c/a\u003e,\n    \u003ca href=\"https://uncertainty-toolbox.github.io/tutorial/\"\u003eTutorials\u003c/a\u003e, and\n    \u003ca href=\"https://uncertainty-toolbox.github.io/docs/\"\u003eDocs\u003c/a\u003e\n    \u0026emsp;\u0026emsp;\u0026nbsp;\u0026nbsp;\n\u003c/p\u003e\n\n\u0026nbsp;\\\n**Uncertainty Toolbox**\n\u003e A Python toolbox for predictive uncertainty quantification, calibration,\n\u003e [metrics, and visualization](#metrics).\\\n\u003e Also: a [glossary of useful terms](docs/glossary.md) and a collection\n\u003e of [relevant papers and references](docs/paper_list.md).\n\n\u0026nbsp;\\\nMany machine learning methods return predictions along with uncertainties of some form,\nsuch as distributions or confidence intervals. This begs the questions: How do we\ndetermine which predictive uncertanties are best? What does it mean to produce a _best_\nor _ideal_ uncertainty? Are our uncertainties accurate and _well calibrated_?\n\nUncertainty Toolbox provides standard metrics to quantify and compare predictive\nuncertainty estimates, gives intuition for these metrics, produces visualizations of\nthese metrics/uncertainties, and implements simple \"re-calibration\" procedures to\nimprove these uncertainties.  This toolbox currently focuses on regression tasks.\n\n\n## Toolbox Contents\n\nUncertainty Toolbox contains:\n* [Glossary](docs/glossary.md) of terms related to predictive uncertainty\n  quantification.\n* [Metrics](#metrics) for assessing quality of predictive uncertainty estimates.\n* [Visualizations](#visualizations) for predictive uncertainty estimates and metrics.\n* [Recalibration](#recalibration) methods for improving the calibration of a predictor.\n* [Paper list](docs/paper_list.md): publications and references on relevant methods and metrics.\n\n\n## Installation\n\nUncertainty Toolbox requires Python 3.6+. For a lightweight installation of the package\nonly, run:\n```bash\npip install uncertainty-toolbox\n```\n\nFor a full installation with examples, tests, and the latest updates, run:\n```bash\ngit clone https://github.com/uncertainty-toolbox/uncertainty-toolbox.git\ncd uncertainty-toolbox\npip install -e . -r requirements/requirements_dev.txt\n```\nNote that the previous command requires pip ≥ 21.3.\n\nTo verify correct installation, you can run the [test suite](tests/) via:\n```bash\nsource shell/run_all_tests.sh\n```\n\n\n## Quick Start\n\n```python\nimport uncertainty_toolbox as uct\n\n# Load an example dataset of 100 predictions, uncertainties, and ground truth values\npredictions, predictions_std, y, x = uct.data.synthetic_sine_heteroscedastic(100)\n\n# Compute all uncertainty metrics\nmetrics = uct.metrics.get_all_metrics(predictions, predictions_std, y)\n```\nThis example computes [metrics](#metrics) for a vector of predicted values\n(`predictions`) and associated uncertainties (`predictions_std`, a vector of standard\ndeviations), taken with respect to a corresponding set of ground truth values `y`.\n\n**Colab notebook:**\nYou can also take a look at [this Colab\nnotebook](https://colab.research.google.com/drive/1lbhwb6MP8FvQh9Q7Jldt4PuPEeSldsG5?usp=sharing),\nwhich walks through a use case of Uncertainty Toolbox.\n\n\n## Metrics\n\nUncertainty Toolbox provides a number of [metrics](uncertainty_toolbox/metrics.py) to\nquantify and compare predictive uncertainty estimates. For example, the\n[`get_all_metrics`](uncertainty_toolbox/metrics.py#L242) function will return:\n1. __average calibration__: _mean absolute calibration error, root mean squared calibration error, miscalibration area._\n2. __adversarial group calibration__: _mean absolute adversarial group calibration error, root mean squared adversarial group calibration error._\n3. __sharpness__: _expected standard deviation._\n4. __proper scoring rules__: _negative log-likelihood, continuous ranked probability score, check score, interval score._\n5. __accuracy__: _mean absolute error, root mean squared error, median absolute error, coefficient of determination, correlation._\n\n\n## Visualizations\n\nThe following plots are a few of the [visualizations](uncertainty_toolbox/viz.py)\nprovided by Uncertainty Toolbox. See [this example](examples/viz_readme_figures.py) for code\nto reproduce these plots.\n\n**Overconfident** (_too little uncertainty_)\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/uncertainty-toolbox/uncertainty-toolbox/master/docs/images/row_1.svg\" alt=\"\" width=\"100%\" align=\"top\"\u003e\n\u003c/p\u003e\n\n**Underconfident** (_too much uncertainty_)\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/uncertainty-toolbox/uncertainty-toolbox/master/docs/images/row_2.svg\" alt=\"\" width=\"100%\" align=\"top\"\u003e\n\u003c/p\u003e\n\n**Well calibrated**\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/uncertainty-toolbox/uncertainty-toolbox/master/docs/images/row_3.svg\" alt=\"\" width=\"100%\" align=\"top\"\u003e\n\u003c/p\u003e\n\nAnd here are a few of the calibration metrics for the above three cases:\n\n|              | Mean absolute calibration error (MACE) | Root mean squared calibration error (RMSCE) | Miscalibration area (MA) |\n| :----- | :-----: | :-----: | :-----: |\n| Overconfident | 0.19429 | 0.21753 | 0.19625 |\n| Underconfident | 0.20692 | 0.23003 | 0.20901 |\n| Well calibrated | 0.00862 | 0.01040 | 0.00865 |\n\n\n## Recalibration\n\nThe following plots show the results of a\n[recalibration](uncertainty_toolbox/recalibration.py) procedure provided by Uncertainty\nToolbox, which transforms a set of predictive uncertainties to improve average\ncalibration. The algorithm is based on isotonic regression, as proposed by [Kuleshov et\nal](docs/paper_list.md#calibration-sharpness-and-recalibration-in-deep-learning).\n\nSee [this example](examples/viz_recalibrate_readme.py) for code to reproduce these plots.\n\n**Recalibrating overconfident predictions**\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://raw.githubusercontent.com/uncertainty-toolbox/uncertainty-toolbox/master/docs/images/before_recal_0.svg\" alt=\"\" width=\"32%\" align=\"top\"\u003e\n\u003cimg src=\"https://raw.githubusercontent.com/uncertainty-toolbox/uncertainty-toolbox/master/docs/images/recal_arrow.svg\" alt=\"\" width=\"22%\" align=\"top\"\u003e\n\u003cimg src=\"https://raw.githubusercontent.com/uncertainty-toolbox/uncertainty-toolbox/master/docs/images/after_recal_0.svg\" alt=\"\" width=\"32%\" align=\"top\"\u003e\n\u003c/p\u003e\n\n|              | Mean absolute calibration error (MACE) | Root mean squared calibration error (RMSCE) | Miscalibration area (MA) |\n| :----- | :-----: | :-----: | :-----: |\n| Before Recalibration | 0.19429 | 0.21753 | 0.19625 |\n| After Recalibration | 0.01124 | 0.02591 | 0.01117 |\n\n**Recalibrating underconfident predictions**\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://raw.githubusercontent.com/uncertainty-toolbox/uncertainty-toolbox/master/docs/images/before_recal_1.svg\" alt=\"\" width=\"32%\" align=\"top\"\u003e\n\u003cimg src=\"https://raw.githubusercontent.com/uncertainty-toolbox/uncertainty-toolbox/master/docs/images/recal_arrow.svg\" alt=\"\" width=\"22%\" align=\"top\"\u003e\n\u003cimg src=\"https://raw.githubusercontent.com/uncertainty-toolbox/uncertainty-toolbox/master/docs/images/after_recal_1.svg\" alt=\"\" width=\"32%\" align=\"top\"\u003e\n\u003c/p\u003e\n\n|              | Mean absolute calibration error (MACE) | Root mean squared calibration error (RMSCE) | Miscalibration area (MA) |\n| :----- | :-----: | :-----: | :-----: |\n| Before Recalibration | 0.20692 | 0.23003 | 0.20901 |\n| After Recalibration | 0.00157 | 0.00205 | 0.00132 |\n\n## Contributing\n\nWe welcome and greatly appreciate contributions from the community! Please see\nour [contributing guidelines](CONTRIBUTING.md) for details on how to help out.\n\n\n## Citation\n\nIf you found this toolbox helpful, please cite the [following\npaper](https://arxiv.org/abs/2109.10254):\n```\n@article{chung2021uncertainty,\n  title={Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification},\n  author={Chung, Youngseog and Char, Ian and Guo, Han and Schneider, Jeff and Neiswanger, Willie},\n  journal={arXiv preprint arXiv:2109.10254},\n  year={2021}\n}\n```\n\nAdditionally, here are papers that led to the development of the toolbox:\n```\n@article{chung2020beyond,\n  title={Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification},\n  author={Chung, Youngseog and Neiswanger, Willie and Char, Ian and Schneider, Jeff},\n  journal={arXiv preprint arXiv:2011.09588},\n  year={2020}\n}\n\n@article{tran2020methods,\n  title={Methods for comparing uncertainty quantifications for material property predictions},\n  author={Tran, Kevin and Neiswanger, Willie and Yoon, Junwoong and Zhang, Qingyang and Xing, Eric and Ulissi, Zachary W},\n  journal={Machine Learning: Science and Technology},\n  volume={1},\n  number={2},\n  pages={025006},\n  year={2020},\n  publisher={IOP Publishing}\n}\n```\n\n\n## Acknowledgments\n\nDevelopment of Uncertainty Toolbox is [supported by](docs/acknowledgments.md) the following organizations.\n\u003cp align=\"top\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/uncertainty-toolbox/uncertainty-toolbox/master/docs/assets/acks_aws.svg\" width=\"7%\"\u003e\n    \u0026nbsp; \u0026nbsp;\n    \u003cimg src=\"https://raw.githubusercontent.com/uncertainty-toolbox/uncertainty-toolbox/master/docs/assets/acks_doe.png\" width=\"8%\"\u003e\n    \u0026nbsp; \u0026nbsp;\n    \u003cimg src=\"https://raw.githubusercontent.com/uncertainty-toolbox/uncertainty-toolbox/master/docs/assets/acks_nsf.png\" width=\"9%\"\u003e\n    \u0026nbsp; \u0026nbsp;\n    \u003cimg src=\"https://raw.githubusercontent.com/uncertainty-toolbox/uncertainty-toolbox/master/docs/assets/cmu_logo.png\" width=\"35%\"\u003e\n    \u0026nbsp; \u0026nbsp;\n\u003c/p\u003e\n\n\u003cp\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/uncertainty-toolbox/uncertainty-toolbox/master/docs/assets/stanford_logo.png\" width=\"20%\"\u003e\n    \u0026nbsp; \u0026nbsp;\n\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Funcertainty-toolbox%2Funcertainty-toolbox","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Funcertainty-toolbox%2Funcertainty-toolbox","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Funcertainty-toolbox%2Funcertainty-toolbox/lists"}