https://github.com/novartis/unique
A Python library for benchmarking uncertainty estimation and quantification methods for Machine Learning models predictions.
https://github.com/novartis/unique
calibration machine-learning uncertainty-quantification
Last synced: 3 months ago
JSON representation
A Python library for benchmarking uncertainty estimation and quantification methods for Machine Learning models predictions.
- Host: GitHub
- URL: https://github.com/novartis/unique
- Owner: Novartis
- License: bsd-3-clause
- Created: 2024-08-19T22:09:07.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2024-10-04T18:16:59.000Z (about 1 year ago)
- Last Synced: 2025-06-29T12:16:42.982Z (3 months ago)
- Topics: calibration, machine-learning, uncertainty-quantification
- Language: Python
- Homepage: https://opensource.nibr.com/UNIQUE
- Size: 6 MB
- Stars: 37
- Watchers: 7
- Forks: 4
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
UNcertaInty QUantification bEnchmark: a Python library for benchmarking uncertainty estimation and quantification methods for Machine Learning models predictions.
[](https://pypi.org/project/unique-uncertainty/)
[](https://pypi.org/project/unique-uncertainty/)
[](https://anaconda.org/conda-forge/unique-uncertainty)
[](https://opensource.org/licenses/BSD-3-Clause)
[](https://doi.org/10.26434/chemrxiv-2024-fmbgk)
[](https://pypi.org/project/unique-uncertainty/)
[](https://anaconda.org/conda-forge/unique-uncertainty)
[](https://opensource.nibr.com/UNIQUE/)
[](https://pypi.org/project/unique-uncertainty/)
## Introduction
`UNIQUE` provides methods for quantifying and evaluating the uncertainty of Machine Learning (ML) models predictions. The library allows to combine and benchmark multiple uncertainty quantification (UQ) methods simultaneously, generates intuitive visualizations, evaluates the goodness of the UQ methods against established metrics, and in general enables the users to get a comprehensive overview of their ML model's performances from an uncertainty quantification perspective.
`UNIQUE` is a model-agnostic tool, meaning that it does not depend on any specific ML model building platform or provides any ML model training functionality. It is lightweight, because it only requires the user to input their model's inputs and predictions.
High-level schema of UNIQUE's components.
## Installation
[](https://pypi.org/project/unique-uncertainty/)
[](https://pypi.org/project/unique-uncertainty/)
[](https://anaconda.org/conda-forge/unique-uncertainty)
[](https://pypi.org/project/unique-uncertainty/)
[](https://anaconda.org/conda-forge/unique-uncertainty)
[](https://pypi.org/project/unique-uncertainty/)
`UNIQUE` is currently compatible with Python 3.8 through 3.12.1. To install the latest release and use the package as is, run the following in a compatible environment of choice:
```bash
pip install unique-uncertainty
```
or:
```bash
conda install -c conda-forge unique-uncertainty
# mamba install -c conda-forge unique-uncertainty
```
Check out the [docs](https://opensource.nibr.com/UNIQUE/installation.html#installation) for more installation instructions.
## Getting Started
Check out the [docs](https://opensource.nibr.com/UNIQUE/getting_started/index.html#getting-started) for a complete set of instructions on how to prepare your data and the possible configurations offered by `UNIQUE`.
## Usage
Finally, once the data and configuration files have been prepared, you can run `UNIQUE` in the following way:
```python
from unique import Pipeline
# Prepare UNIQUE pipeline
pipeline = Pipeline.from_config("/path/to/config.yaml")
# Run UNIQUE pipeline
uq_methods_outputs, uq_evaluation_outputs = pipeline.fit()
# Returns: (Dict[str, np.ndarray], Dict[str, pd.DataFrame])
```
Fitting the `Pipeline` will return two dictionaries:
- `uq_methods_outputs`: contains each UQ method's name (as in "UQ_Method_Name[Input_Name(s)]") and computed UQ values.
- `uq_evaluation_outputs`: contains, for each evaluation type (ranking-based, proper scoring rules, and calibration-based), the evaluation metrics outputs for all the corresponding UQ methods organized in `pd.DataFrame`.
Additionally, `UNIQUE` also generates graphical outputs in the form of tables and evaluation plots (if `display_outputs` is enabled and the code is running in a JupyterNotebook cell).
### Examples
For more hands-on examples and detailed usage, check out some of the examples in the [docs](https://opensource.nibr.com/UNIQUE/examples/index.html#examples).
## Deep Dive
Check out the [docs](https://opensource.nibr.com/UNIQUE/indepth/index.html#deep-dive) for an in-depth overview of `UNIQUE`'s concepts, functionalities, outputs, and references.
## Contributing
Any and all contributions and suggestions from the community are more than welcome and highly appreciated. If you wish to help us out in making `UNIQUE` even better, please check out our [contributing guidelines](./CONTRIBUTING.md).
Please note that we have a [Code of Conduct](./CODE_OF_CONDUCT.md) in place to ensure a positive and inclusive community environment. By participating in this project, you agree to abide by its terms.
## License
[](https://opensource.org/licenses/BSD-3-Clause)
`UNIQUE` is licensed under the BSD 3-Clause License. See the [LICENSE](./LICENSE.md) file.
## Cite Us
[](https://doi.org/10.26434/chemrxiv-2024-fmbgk)
If you find `UNIQUE` helpful for your work and/or research, please consider citing our work:
```bibtex
@misc{lanini2024unique,
title={UNIQUE: A Framework for Uncertainty Quantification Benchmarking},
author={Lanini, Jessica and Huynh, Minh Tam Davide and Scebba, Gaetano and Schneider, Nadine and Rodr{\'\i}guez-P{\'e}rez, Raquel},
year={2024},
doi={https://doi.org/10.26434/chemrxiv-2024-fmbgk},
}
```
## Contacts & Acknowledgements
For any questions or further details about the project, please get in touch with any of the following contacts:
* **[Jessica Lanini](mailto:jessica.lanini@novartis.com?subject=UNIQUE)**
* **[Minh Tam Davide Huynh](https://github.com/mtdhuynh)**
* **[Gaetano Scebba](mailto:gaetano.scebba@novartis.com?subject=UNIQUE)**
* **[Nadine Schneider](mailto:nadine-1.schneider@novartis.com?subject=UNIQUE)**
* **[Raquel Rodríguez-Pérez](mailto:raquel.rodriguez_perez@novartis.com?subject=UNIQUE)**
