{"id":13487593,"url":"https://github.com/awslabs/fortuna","last_synced_at":"2025-03-27T22:31:42.579Z","repository":{"id":64995201,"uuid":"567281014","full_name":"awslabs/fortuna","owner":"awslabs","description":"A Library for Uncertainty Quantification.","archived":false,"fork":false,"pushed_at":"2025-03-06T04:02:20.000Z","size":6480,"stargazers_count":905,"open_issues_count":6,"forks_count":48,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-03-14T06:07:38.704Z","etag":null,"topics":["ai","bayesian-inference","calibration","conformal-prediction","deep-learning","flax","jax","machine-learning","ml","model-calibration","neural-networks","uncertainty","uncertainty-calibration","uncertainty-estimation","uncertainty-quantification"],"latest_commit_sha":null,"homepage":"https://aws-fortuna.readthedocs.io/en/latest/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/awslabs.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-17T13:11:46.000Z","updated_at":"2025-03-13T09:14:57.000Z","dependencies_parsed_at":"2023-01-12T08:15:32.856Z","dependency_job_id":"b044a08e-3333-45b0-94c7-b0b219411bc3","html_url":"https://github.com/awslabs/fortuna","commit_stats":{"total_commits":228,"total_committers":10,"mean_commits":22.8,"dds":0.5789473684210527,"last_synced_commit":"0fdbb3e03907244c500838e48068538244688bea"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/awslabs%2Ffortuna","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/awslabs%2Ffortuna/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/awslabs%2Ffortuna/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/awslabs%2Ffortuna/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/awslabs","download_url":"https://codeload.github.com/awslabs/fortuna/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245935686,"owners_count":20696683,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","bayesian-inference","calibration","conformal-prediction","deep-learning","flax","jax","machine-learning","ml","model-calibration","neural-networks","uncertainty","uncertainty-calibration","uncertainty-estimation","uncertainty-quantification"],"created_at":"2024-07-31T18:01:00.985Z","updated_at":"2025-03-27T22:31:42.557Z","avatar_url":"https://github.com/awslabs.png","language":"Python","funding_links":[],"categories":["Other Toolboxes","Python","Libraries"],"sub_categories":[],"readme":"Fortuna\n#######\n\n.. image:: https://img.shields.io/pypi/status/Fortuna\n    :target: https://img.shields.io/pypi/status/Fortuna\n    :alt: PyPI - Status\n.. image:: https://img.shields.io/pypi/dm/aws-fortuna\n    :target: https://pypistats.org/packages/aws-fortuna\n    :alt: PyPI - Downloads\n.. image:: https://img.shields.io/pypi/v/aws-fortuna\n    :target: https://img.shields.io/pypi/v/aws-fortuna\n    :alt: PyPI - Version\n.. image:: https://img.shields.io/github/license/awslabs/Fortuna\n    :target: https://github.com/awslabs/Fortuna/blob/main/LICENSE\n    :alt: License\n.. image:: https://readthedocs.org/projects/aws-fortuna/badge/?version=latest\n    :target: https://aws-fortuna.readthedocs.io\n    :alt: Documentation Status\n\nA Library for Uncertainty Quantification\n========================================\nProper estimation of predictive uncertainty is fundamental in applications that involve critical decisions.\nUncertainty can be used to assess reliability of model predictions, trigger human intervention,\nor decide whether a model can be safely deployed in the wild.\n\nFortuna is a library for uncertainty quantification that makes it easy for users to run benchmarks and bring uncertainty to production systems.\nFortuna provides calibration and conformal methods starting from pre-trained models written in any framework,\nand it further supports several Bayesian inference methods starting from deep learning models written in `Flax \u003chttps://flax.readthedocs.io/en/latest/index.html\u003e`_.\nThe language is designed to be intuitive for practitioners unfamiliar with uncertainty quantification,\nand is highly configurable.\n\nCheck the `documentation \u003chttps://aws-fortuna.readthedocs.io/en/latest/\u003e`_ for a quickstart, examples and references.\n\nUsage modes\n===========\nFortuna offers three different usage modes:\n`From uncertainty estimates \u003chttps://github.com/awslabs/fortuna#from-uncertainty-estimates\u003e`_,\n`From model outputs \u003chttps://github.com/awslabs/fortuna#from-model-outputs\u003e`_ and\n`From Flax models \u003chttps://github.com/awslabs/fortuna#from-flax-models\u003e`_.\nThese serve users according to the constraints dictated by their own applications.\nTheir pipelines are depicted in the following figure, each starting from one of the green panels.\n\n.. image:: https://github.com/awslabs/fortuna/raw/main/docs/source/_static/pipeline.png\n    :target: https://github.com/awslabs/fortuna/raw/main/docs/source/_static/pipeline.png\n\nFrom uncertainty estimates\n---------------------------\nStarting from uncertainty estimates has minimal compatibility requirements and it is the quickest level of interaction with the library.\nThis usage mode offers conformal prediction methods for both classification and regression.\nThese take uncertainty estimates in input,\nand return rigorous sets of predictions that retain a user-given level of probability.\nIn one-dimensional regression tasks, conformal sets may be thought as calibrated versions of confidence or credible intervals.\n\nMind that if the uncertainty estimates that you provide in inputs are inaccurate,\nconformal sets might be large and unusable.\nFor this reason, if your application allows it,\nplease consider the `From model outputs \u003chttps://github.com/awslabs/fortuna#from-model-outputs\u003e`_ and\n`From Flax models \u003chttps://github.com/awslabs/fortuna#from-flax-models\u003e`_ usage modes.\n\n**Example.** Suppose you want to calibrate credible intervals with coverage error :code:`error`,\neach corresponding to a different test input variable.\nWe assume that credible intervals are passed as arrays of lower and upper bounds,\nrespectively :code:`test_lower_bounds` and :code:`test_upper_bounds`.\nYou also have lower and upper bounds of credible intervals computed for several validation inputs,\nrespectively :code:`val_lower_bounds` and :code:`val_upper_bounds`.\nThe corresponding array of validation targets is denoted by :code:`val_targets`.\nThe following code produces *conformal prediction intervals*,\ni.e. calibrated versions of you test credible intervals.\n\n.. code-block:: python\n\n from fortuna.conformal import QuantileConformalRegressor\n conformal_intervals = QuantileConformalRegressor().conformal_interval(\n      val_lower_bounds=val_lower_bounds, val_upper_bounds=val_upper_bounds,\n      test_lower_bounds=test_lower_bounds, test_upper_bounds=test_upper_bounds,\n      val_targets=val_targets, error=error)\n\nFrom model outputs\n------------------\nStarting from model outputs assumes you have already trained a model in some framework,\nand arrive to Fortuna with model outputs in :code:`numpy.ndarray` format for each input data point.\nThis usage mode allows you to calibrate your model outputs, estimate uncertainty,\ncompute metrics and obtain conformal sets.\n\nCompared to the `From uncertainty estimates \u003chttps://github.com/awslabs/fortuna#from-uncertainty-estimates\u003e`_ usage mode,\nthis one offers better control,\nas it can make sure uncertainty estimates have been appropriately calibrated.\nHowever, if the model had been trained with classical methods,\nthe resulting quantification of model (a.k.a. epistemic) uncertainty may be poor.\nTo mitigate this problem, please consider the `From Flax models \u003chttps://github.com/awslabs/fortuna#from-flax-models\u003e`_\nusage mode.\n\n**Example.**\nSuppose you have validation and test model outputs,\nrespectively :code:`val_outputs` and :code:`test_outputs`.\nFurthermore, you have some arrays of validation and target variables,\nrespectively :code:`val_targets` and :code:`test_targets`.\nThe following code provides a minimal classification example to get calibrated predictive entropy estimates.\n\n.. code-block:: python\n\n  from fortuna.output_calib_model import OutputCalibClassifier\n  calib_model = OutputCalibClassifier()\n  status = calib_model.calibrate(outputs=val_outputs, targets=val_targets)\n  test_entropies = calib_model.predictive.entropy(outputs=test_outputs)\n\nFrom Flax models\n--------------------------\nStarting from Flax models has higher compatibility requirements than the\n`From uncertainty estimates \u003chttps://github.com/awslabs/fortuna#from-uncertainty-estimates\u003e`_\nand `From model outputs \u003chttps://github.com/awslabs/fortuna#from-model-outputs\u003e`_ usage modes,\nas it requires deep learning models written in `Flax \u003chttps://flax.readthedocs.io/en/latest/index.html\u003e`_.\nHowever, it enables you to replace standard model training with scalable Bayesian inference procedures,\nwhich may significantly improve the quantification of predictive uncertainty.\n\n**Example.** Suppose you have a Flax classification deep learning model :code:`model` from inputs to logits, with output\ndimension given by :code:`output_dim`. Furthermore,\nyou have some training, validation and calibration TensorFlow data loader :code:`train_data_loader`, :code:`val_data_loader`\nand :code:`test_data_loader`, respectively.\nThe following code provides a minimal classification example to get calibrated probability estimates.\n\n.. code-block:: python\n\n  from fortuna.data import DataLoader\n  train_data_loader = DataLoader.from_tensorflow_data_loader(train_data_loader)\n  calib_data_loader = DataLoader.from_tensorflow_data_loader(val_data_loader)\n  test_data_loader = DataLoader.from_tensorflow_data_loader(test_data_loader)\n\n  from fortuna.prob_model import ProbClassifier\n  prob_model = ProbClassifier(model=model)\n  status = prob_model.train(train_data_loader=train_data_loader, calib_data_loader=calib_data_loader)\n  test_means = prob_model.predictive.mean(inputs_loader=test_data_loader.to_inputs_loader())\n\n\nInstallation\n============\n**NOTE:** Before installing Fortuna, you are required to `install JAX \u003chttps://github.com/google/jax#installation\u003e`_ in your virtual environment.\n\nYou can install Fortuna by typing\n\n.. code-block::\n\n    pip install aws-fortuna\n\nAlternatively, you can build the package using `Poetry \u003chttps://python-poetry.org/docs/\u003e`_.\nIf you choose to pursue this way, first install Poetry and add it to your PATH\n(see `here \u003chttps://python-poetry.org/docs/#installation\u003e`_). Then type\n\n.. code-block::\n\n    poetry install\n\nAll the dependencies will be installed at their required versions. Consider adding the following flags to the command above:\n\n- :code:`-E transformers` if you want to use models and datasets from `Hugging Face \u003chttps://huggingface.co/\u003e`_.\n- :code:`-E sagemaker` if you want to install the dependencies necessary to run Fortuna on Amazon SageMaker.\n- :code:`-E docs` if you want to install Sphinx dependencies to build the documentation.\n- :code:`-E notebooks` if you want to work with Jupyter notebooks.\n\nFinally, you can either access the virtualenv that Poetry created by typing :code:`poetry shell`,\nor execute commands within the virtualenv using the :code:`run` command, e.g. :code:`poetry run python`.\n\nExamples\n========\nSeveral usage examples are found in the\n`/examples \u003chttps://github.com/awslabs/fortuna/tree/main/examples\u003e`_\ndirectory.\n\nAmazon SageMaker integration\n============================\nWe offer a simple pipeline that allows you to run Fortuna on Amazon SageMaker with minimal effort.\n\n1. Create an AWS account - it is free! Store the account ID and the region where you want to launch training jobs.\n\n2. First, `update your local AWS credentials \u003chttps://docs.aws.amazon.com/cli/latest/userguide/cli-authentication-short-term.html\u003e`_.\n   Then you need to build and `push a Docker image to an Amazon ECR repository \u003chttps://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html\u003e`_.\n   This `script \u003chttps://github.com/awslabs/fortuna/tree/main/fortuna/docker/build_and_push.sh\u003e`_ will help you doing so -\n   it will require your AWS account ID and region. If you need other packages to be included in your Docker image,\n   you should consider customize the `Dockerfile \u003chttps://github.com/awslabs/fortuna/tree/main/fortuna/docker/Dockerfile\u003e`_.\n   NOTE: the script has been tested on a M1 MacOS.\n   It is possible that different operating systems will need small modifications.\n\n3. Create an `S3 bucket \u003chttps://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html\u003e`_.\n   You will need this to dump the results from your training jobs on Amazon Sagemaker.\n\n4. Write a configuration `yaml` file. This will include your AWS details, the path to the entrypoint script that you want\n   to run on Amazon SageMaker, the arguments to pass to the script, the path to the S3 bucket where you want to dump\n   the results, the metrics to monitor, and more.\n   Check `this file \u003chttps://github.com/awslabs/fortuna/tree/main/benchmarks/transformers/sagemaker_entrypoints/prob_model_text_classification_config/default.yaml\u003e`_ for an example.\n\n5. Finally, given :code:`config_dir`, that is the absolute path to the main configuration directory,\n   and :code:`config_filename`, that is the name of the main configuration file (without .yaml extension),\n   enter Python and run the following:\n\n.. code-block:: python\n\n    from fortuna.sagemaker import run_training_job\n    run_training_job(config_dir=config_dir, config_filename=config_filename)\n\n\nMaterial\n========\n- `AWS launch blog post \u003chttps://aws.amazon.com/blogs/machine-learning/introducing-fortuna-a-library-for-uncertainty-quantification/\u003e`_\n- `Fortuna: A Library for Uncertainty Quantification in Deep Learning [arXiv paper] \u003chttps://arxiv.org/abs/2302.04019\u003e`_\n\nCiting Fortuna\n==============\nTo cite Fortuna:\n\n.. code-block::\n\n    @article{detommaso2023fortuna,\n      title={Fortuna: A Library for Uncertainty Quantification in Deep Learning},\n      author={Detommaso, Gianluca and Gasparin, Alberto and Donini, Michele and Seeger, Matthias and Wilson, Andrew Gordon and Archambeau, Cedric},\n      journal={arXiv preprint arXiv:2302.04019},\n      year={2023}\n    }\n\nContributing\n============\nIf you wish to contribute to the project, please refer to our `contribution guidelines \u003chttps://github.com/awslabs/fortuna/blob/main/CONTRIBUTING.md\u003e`_.\n\n\nLicense\n=======\nThis project is licensed under the Apache-2.0 License.\nSee `LICENSE \u003chttps://github.com/awslabs/fortuna/blob/main/LICENSE\u003e`_ for more information.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fawslabs%2Ffortuna","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fawslabs%2Ffortuna","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fawslabs%2Ffortuna/lists"}