{"id":27248562,"url":"https://github.com/deel-ai/oodeel","last_synced_at":"2025-09-03T00:49:32.004Z","repository":{"id":106288647,"uuid":"585229209","full_name":"deel-ai/oodeel","owner":"deel-ai","description":"Simple, compact, and hackable post-hoc deep OOD detection for already trained tensorflow or pytorch image classifiers.","archived":false,"fork":false,"pushed_at":"2025-07-28T09:09:21.000Z","size":107084,"stargazers_count":59,"open_issues_count":15,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-09-03T00:49:30.070Z","etag":null,"topics":["deep-neural-networks","out-of-distribution-detection","post-hoc-analysis","pytorch","robustness","tensorflow"],"latest_commit_sha":null,"homepage":"https://deel-ai.github.io/oodeel/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deel-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":"GOVERNANCE.md","roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-01-04T16:36:12.000Z","updated_at":"2025-08-14T21:02:42.000Z","dependencies_parsed_at":"2024-07-16T20:01:01.982Z","dependency_job_id":"68b97212-5bc6-4a44-97c6-96c7bde05454","html_url":"https://github.com/deel-ai/oodeel","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/deel-ai/oodeel","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deel-ai%2Foodeel","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deel-ai%2Foodeel/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deel-ai%2Foodeel/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deel-ai%2Foodeel/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deel-ai","download_url":"https://codeload.github.com/deel-ai/oodeel/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deel-ai%2Foodeel/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273372600,"owners_count":25093713,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-02T02:00:09.530Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-neural-networks","out-of-distribution-detection","post-hoc-analysis","pytorch","robustness","tensorflow"],"created_at":"2025-04-10T23:40:09.279Z","updated_at":"2025-09-03T00:49:31.952Z","avatar_url":"https://github.com/deel-ai.png","language":"Python","funding_links":[],"categories":["Libraries"],"sub_categories":[],"readme":"\n\u003c!-- Banner section --\u003e\n\u003cdiv align=\"center\"\u003e\n        \u003cpicture\u003e\n                \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"docs/assets/oodeel_dark.png\"\u003e\n                \u003csource media=\"(prefers-color-scheme: light)\" srcset=\"docs/assets/oodeel_light.png\"\u003e\n                \u003cimg src=\"docs/assets/oodeel_light.png\" alt=\"Library Banner\"\u003e\n        \u003c/picture\u003e\n\u003c/div\u003e\n\u003cbr\u003e\n\n\u003c!-- Badge section --\u003e\n\u003cdiv align=\"center\"\u003e\n    \u003ca href=\"#\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/python-3.8%2B-blue\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/deel-ai/oodeel/actions/workflows/python-linters.yml\"\u003e\n        \u003cimg alt=\"Flake8\" src=\"https://github.com/deel-ai/oodeel/actions/workflows/python-linters.yml/badge.svg\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/deel-ai/oodeel/actions/workflows/python-tests-tf.yml\"\u003e\n        \u003cimg alt=\"Tests tf\" src=\"https://github.com/deel-ai/oodeel/actions/workflows/python-tests-tf.yml/badge.svg\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/deel-ai/oodeel/actions/workflows/python-tests-torch.yml\"\u003e\n        \u003cimg alt=\"Tests torch\" src=\"https://github.com/deel-ai/oodeel/actions/workflows/python-tests-torch.yml/badge.svg\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/deel-ai/oodeel/actions/workflows/python-coverage-shield.yml\"\u003e\n        \u003cimg alt=\"Coverage\" src=\"https://github.com/deel-ai/oodeel/raw/gh-shields/coverage.svg\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/deel-ai/oodeel/blob/master/LICENSE\"\u003e\n        \u003cimg alt=\"License MIT\" src=\"https://img.shields.io/badge/License-MIT-efefef\"\u003e\u003c/a\u003e\n\u003c/div\u003e\n\u003cbr\u003e\n\n\u003c!-- Short description of your library --\u003e\n\n\u003cb\u003eOodeel\u003c/b\u003e is a library that performs post-hoc deep OOD (Out-of-Distribution) detection on already trained neural network image classifiers. The philosophy of the library is to favor quality over quantity and to foster easy adoption. As a result, we provide a simple, compact and easily customizable API and carefully integrate and test each proposed baseline into a coherent framework that is designed to enable their use in tensorflow **and** pytorch. You can find the documentation [here](https://deel-ai.github.io/oodeel/).\n\n```python\nfrom oodeel.methods import MLS\n\nmls = MLS()\nmls.fit(model) # A tensorflow or torch model\nscores, info = mls.score(ds) # ds is a tf.data.Dataset or a torch.DataLoader\n```\n\n# Table of contents\n\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Tutorials](#tutorials)\n- [What's Included](#whats-included)\n- [Development roadmap](#development-roadmap)\n- [Contributing](#contributing)\n- [See Also](#see-also)\n- [Acknowledgments](#acknowledgments)\n- [Creators](#creators)\n- [Citation](#citation)\n- [License](#license)\n\n# Installation\n\nInstallation can be done using:\n\n```bash\npip install oodeel\n```\n\noodeel requires either `tensorflow` or `pytorch` to be already installed (it will not install them automatically not to mess-up with existing installations). It is regularly tested with:\n\n|Python version|Pytorch version|Tensorflow version|\n|---|---|---|\n|`3.8`| `1.11`| `2.5`|\n|`3.9`|`1.13` | `2.8`|\n|`3.10`| `2.00` | `2.11`|\n\n# Quick Start\n\nNow that *oodeel* is installed, here are some basic examples of what you can do with the available modules. See also the notebooks directory for more advanced examples.\n\n### For benchmarking with one dataset as in-distribution and another as out-of-distribution\n\nLoad in-distribution and out-of-distribution datasets.\n\n```python\nfrom oodeel.datasets import load_data_handler\n\ndata_handler = load_data_handler(\"torch\")\n# use backend=\"tensorflow\" if using tensorflow\n# Do not forget to adapt load_kwargs in that case\n\n# Load in-distribution dataset: CIFAR-10\nds_in = data_handler.load_dataset(\n    \"CIFAR10\", load_kwargs={\"root\": data_path, \"train\": False, \"download\": True}\n)\n# Load out-of-distribution dataset: SVHN\nds_out = data_handler.load_dataset(\n    \"SVHN\", load_kwargs={\"root\": data_path, \"split\": \"test\", \"download\": True}\n)\n\n# Prepare datasets for forward (requires appropriate preprocess_fn e.g. input normalization)\nds_in = data_handler.prepare(\n    ds_in, batch_size, preprocess_fn, columns=[\"input\", \"label\"]\n)\nds_out = data_handler.prepare(\n    ds_out, batch_size, preprocess_fn, columns=[\"input\", \"label\"]\n)\n```\n\n### For benchmarking with a classes subset as in-distribution and another classes subset as out-of-distribution\n\nLoad a dataset and split it into an in-distribution dataset and ou-of-distribution dataset depending on its label values (a common practice of anomaly detection and open set recognition).\n\n```python\nfrom oodeel.datasets import load_data_handler\n\ndata_handler = load_data_handler(\"torch\") # use backend=\"tensorflow\" if using tensorflow\n\n# Load dataset to split: CIFAR-10\nds_test = data_handler.load_dataset(\n    \"CIFAR10\", load_kwargs={\"root\": data_path, \"train\": False, \"download\": True}\n)\n\nin_labels = [0, 1, 2, 3, 4]\nds_in, ds_out = data_handler.split_by_class(ds_test, in_labels)\n\n# Prepare datasets for forward (requires appropriate preprocess_fn e.g. input normalization)\nds_in = data_handler.prepare(\n    ds_in, batch_size, preprocess_fn, columns=[\"input\", \"label\"]\n)\nds_out = data_handler.prepare(\n    ds_out, batch_size, preprocess_fn, columns=[\"input\", \"label\"]\n)\n```\n### Run an OOD method\n\nLoad an OOD method and use it on an already-trained model\n\n```python\nfrom oodeel.methods import MLS\n\nmls = MLS()\nmls.fit(model) # Requires a pretrained model\n\n# info_X contains model predictions and labels if available\nscores_in, info_in = mls.score(ds_in)\nscores_out, info_out = mls.score(ds_out)\n```\n\nEvaluate the method\n\n```python\nfrom oodeel.eval.metrics import bench_metrics\n\nmetrics = bench_metrics(\n    (scores_in, scores_out),\n    metrics = [\"auroc\", \"fpr95tpr\"],\n    )\n```\n\n### And visualize the results!\n\n2D t-SNE (3D is also available).\n\n```python\nplot_2D_features(\n    model=model,\n    in_dataset=ds_in,\n    out_dataset=ds_out,\n    output_layer_id=-2,\n)\n```\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/assets/tsne.png\" alt=\"TSNE\" /\u003e\n\u003c/p\u003e\n\nClassical histograms and AUROC curve.\n```python\nplot_ood_scores(scores_in, scores_out, log_scale=False)\nplot_roc_curve(scores_in, scores_out)\n```\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/assets/auroc.png\" alt=\"AUROC\" /\u003e\n\u003c/p\u003e\n\n# Tutorials\n\nWe propose some tutorials to get familiar with the library and its API. See the Tutorial section of the [doc](https://deel-ai.github.io/oodeel/)\n\n\n# What's Included\n\nThe library is based on a class, `OODBaseDetector`, that fits a model and then scores new samples. Some baselines use extra data, so `OODBaseDetector` can also fit additional data if needed. The library uses `OODDataset` to properly load data from different sources and prepare it for OOD detection. It can perform OOD-specific operations like adding extra OOD data for tuning with Outlier Exposure or filters according to label values for anomaly detection or open set recognition benchmarks.\n\nCurrently, **oodeel** includes the following baselines:\n\n\n\n|  Name   |  Link   | Venue | Status |\n| --- | --- | --- | --- |\n| MLS | [Open-Set Recognition: a Good Closed-Set Classifier is All You Need?](http://arxiv.org/abs/2110.06207) | ICLR 2022 | avail [tensorflow \u0026 torch](docs/pages/getting_started.ipynb) |\n| MSP | [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](http://arxiv.org/abs/1610.02136) | ICLR 2017 | avail [tensorflow \u0026 torch](docs/pages/getting_started.ipynb)|\n| Mahalanobis | [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](http://arxiv.org/abs/1807.03888) | NeurIPS 2018 | avail [tensorflow](docs/notebooks/tensorflow/demo_mahalanobis_tf.ipynb) or  [torch](docs/notebooks/torch/demo_mahalanobis_torch.ipynb)|\n| Energy | [Energy-based Out-of-distribution Detection](http://arxiv.org/abs/2010.03759) | NeurIPS 2020 |avail [tensorflow](docs/notebooks/tensorflow/demo_energy_tf.ipynb) or  [torch](docs/notebooks/torch/demo_energy_torch.ipynb)  |\n| Odin | [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](http://arxiv.org/abs/1706.02690) | ICLR 2018 | avail [tensorflow](docs/notebooks/tensorflow/demo_odin_tf.ipynb) or  [torch](docs/notebooks/torch/demo_odin_torch.ipynb) |\n| DKNN | [Out-of-Distribution Detection with Deep Nearest Neighbors](http://arxiv.org/abs/2204.06507) | ICML 2022 | avail [tensorflow](docs/notebooks/tensorflow/demo_dknn_tf.ipynb) or  [torch](docs/notebooks/torch/demo_dknn_torch.ipynb) |\n| VIM | [ViM: Out-Of-Distribution with Virtual-logit Matching](http://arxiv.org/abs/2203.10807) | CVPR 2022 |avail [tensorflow](docs/notebooks/tensorflow/demo_vim_tf.ipynb) or  [torch](docs/notebooks/torch/demo_vim_torch.ipynb)  |\n| Entropy | [Likelihood Ratios for Out-of-Distribution Detection](https://proceedings.neurips.cc/paper/2019/hash/1e79596878b2320cac26dd792a6c51c9-Abstract.html) | NeurIPS 2019 |avail [tensorflow](docs/notebooks/tensorflow/demo_entropy_tf.ipynb) or  [torch](docs/notebooks/torch/demo_entropy_torch.ipynb)  |\n| ReAct | [ReAct: Out-of-distribution Detection With Rectified Activations](http://arxiv.org/abs/2111.12797) | NeurIPS 2021 | avail [tensorflow](docs/notebooks/tensorflow/demo_react_tf.ipynb) or  [torch](docs/notebooks/torch/demo_react_torch.ipynb) |\n| Gram | [Detecting Out-of-Distribution Examples with Gram Matrices](https://proceedings.mlr.press/v119/sastry20a.html) | ICML 2020 | avail [tensorflow](docs/notebooks/tensorflow/demo_gram_tf.ipynb) or  [torch](docs/notebooks/torch/demo_gram_torch.ipynb) |\n| GEN | [GEN: Pushing the Limits of Softmax-Based Out-of-Distribution Detection](https://openaccess.thecvf.com/content/CVPR2023/html/Liu_GEN_Pushing_the_Limits_of_Softmax-Based_Out-of-Distribution_Detection_CVPR_2023_paper.html) | CVPR 2023 | avail [tensorflow](docs/notebooks/tensorflow/demo_gen_tf.ipynb) or [torch](docs/notebooks/torch/demo_gen_torch.ipynb) |\n| RMDS | [A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection](https://arxiv.org/abs/2106.09022) | preprint | avail [tensorflow](docs/notebooks/tensorflow/demo_rmds_tf.ipynb) or [torch](docs/notebooks/torch/demo_rmds_torch.ipynb) |\n| SHE | [Out-of-Distribution Detection based on In-Distribution Data Patterns Memorization with Modern Hopfield Energy](https://openreview.net/forum?id=KkazG4lgKL) | ICLR 2023 | avail [tensorflow](docs/notebooks/tensorflow/demo_she_tf.ipynb) or [torch](docs/notebooks/torch/demo_she_torch.ipynb) |\n| ASH | [Extremely Simple Activation Shaping for Out-of-Distribution Detection](https://arxiv.org/abs/2209.09858) | ICLR 2023 | avail [tensorflow](docs/notebooks/tensorflow/demo_ash_tf.ipynb) or [torch](docs/notebooks/torch/demo_ash_torch.ipynb) |\n| SCALE | [Scaling for Training Time and Post-hoc Out-of-distribution Detection Enhancement](https://arxiv.org/abs/2310.00227) | ICLR 2024 | avail [tensorflow](docs/notebooks/tensorflow/demo_scale_tf.ipynb) or [torch](docs/notebooks/torch/demo_scale_torch.ipynb) |\n\n\n\n**Oodeel** also includes standard training functions with data augmentation and learning rate scheduler for toy convnet models or models from `keras.applications` in [tf_training_tools.py](https://github.com/deel-ai/oodeel/tree/master/oodeel/utils/tf_training_tools.py) and `torchvision.models` in [torch_training_tools.py](https://github.com/deel-ai/oodeel/tree/master/oodeel/utils/torch_training_tools.py) files. These functions come in handy for benchmarks like *leave-k-classes-out* that requires retraining models on a subset of dataset classes.\n# Development Roadmap\n\n- [ ] More baselines!\n- [x] A module for thorough visualizations (result plots and feature space visualizations)\n- [ ] Integrate model loading and uploading with [hugginface's transformers](https://huggingface.co/docs/transformers/index) library for pretraining\n- [ ] Extend the library to more diverse tasks like object detection, segmentation, NLP ...\n- [ ] Towards OOD Generalization?\n\n# Contributing\n\nFeel free to propose your ideas or come and contribute with us on the oodeel toolbox! We have a specific document where we describe in a simple way how to make your first pull request: [just here](CONTRIBUTING.md).\n\n# See Also\n\nOther great tools in the field of OOD:\n\n- [OpenOOD](https://github.com/Jingkang50/OpenOOD): Benchmarking Generalized Out-of-Distribution Detection\n- [Pytorch-OOD](https://github.com/kkirchheim/pytorch-ood): Out-of-Distribution (OOD) Detection with Deep Neural Networks based on PyTorch.\n- [ADBench](https://github.com/Minqi824/ADBench): Official Implement of \"ADBench: Anomaly Detection Benchmark\".\n- [PyOD](https://github.com/yzhao062/pyod): A Comprehensive and Scalable Python Library for Outlier Detection (Anomaly Detection)\n- [Anomalib](https://github.com/openvinotoolkit/anomalib): An anomaly detection library comprising state-of-the-art algorithms and features such as experiment management, hyper-parameter optimization, and edge inference.\n\nMore from the DEEL project:\n\n- [Xplique](https://github.com/deel-ai/xplique) a Python library exclusively dedicated to explaining neural networks.\n- [deel-lip](https://github.com/deel-ai/deel-lip) a Python library for training k-Lipschitz neural networks on TF.\n- [Influenciae](https://github.com/deel-ai/influenciae) Python toolkit dedicated to computing influence values for the discovery of potentially problematic samples in a dataset.\n- [deel-torchlip](https://github.com/deel-ai/deel-torchlip) a Python library for training k-Lipschitz neural networks on PyTorch.\n- [DEEL White paper](https://arxiv.org/abs/2103.10529) a summary of the DEEL team on the challenges of certifiable AI and the role of data quality, representativity and explainability for this purpose.\n\n# Acknowledgments\n\n\u003cdiv align=\"right\"\u003e\n  \u003cpicture\u003e\n    \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"docs/assets/deel_dark.png\"  width=\"25%\" align=\"right\"\u003e\n    \u003csource media=\"(prefers-color-scheme: light)\" srcset=\"docs/assets/deel_light.png\"  width=\"25%\" align=\"right\"\u003e\n    \u003cimg src=\"docs/assets/deel_dark.png\" alt=\"DEEL Logo\" width=\"25%\" align=\"right\"\u003e\n  \u003c/picture\u003e\n\u003c/div\u003e\nThis project received funding from the French ”Investing for the Future – PIA3” program within the Artificial and Natural Intelligence Toulouse Institute (ANITI). The authors gratefully acknowledge the support of the \u003ca href=\"https://www.deel.ai/\"\u003e DEEL \u003c/a\u003e, a research project jointly conducted in France and Quebec.\n\n# Creators\n\nThe library was created by Paul Novello to streamline DEEL research on post-hoc deep OOD methods and foster their adoption by DEEL industrial partners. He was soon joined by Yann Pequignot, Yannick Prudent, Corentin Friedrich and Matthieu Le Goff.\n\n# Citation\n\nIf you use OODEEL for your research project, please consider citing:\n```\n@misc{oodeel,\n  author = {Novello, Paul and Prudent, Yannick and Friedrich, Corentin and Pequignot, Yann and Le Goff, Matthieu},\n  title = {OODEEL, a simple, compact, and hackable post-hoc deep OOD detection for already trained tensorflow or pytorch image classifiers.},\n  year = {2023},\n  publisher = {GitHub},\n  journal = {GitHub repository},\n  howpublished = {\\url{https://github.com/deel-ai/oodeel}},\n}\n```\n# License\n\nThe package is released under [MIT license](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeel-ai%2Foodeel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeel-ai%2Foodeel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeel-ai%2Foodeel/lists"}