{"id":26696720,"url":"https://github.com/cleanlab/ood-detection-benchmarks","last_synced_at":"2025-04-13T03:28:18.639Z","repository":{"id":56708814,"uuid":"495604378","full_name":"cleanlab/ood-detection-benchmarks","owner":"cleanlab","description":"Evaluation of algorithms to detect out-of-distribution data","archived":false,"fork":false,"pushed_at":"2022-10-13T00:16:10.000Z","size":11468,"stargazers_count":6,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-26T20:34:48.873Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cleanlab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-05-23T23:28:33.000Z","updated_at":"2024-03-27T13:33:25.000Z","dependencies_parsed_at":"2023-01-19T22:03:44.042Z","dependency_job_id":null,"html_url":"https://github.com/cleanlab/ood-detection-benchmarks","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cleanlab%2Food-detection-benchmarks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cleanlab%2Food-detection-benchmarks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cleanlab%2Food-detection-benchmarks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cleanlab%2Food-detection-benchmarks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cleanlab","download_url":"https://codeload.github.com/cleanlab/ood-detection-benchmarks/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248659182,"owners_count":21141091,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-26T20:27:28.841Z","updated_at":"2025-04-13T03:28:18.614Z","avatar_url":"https://github.com/cleanlab.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Out-of-Distribution (OOD) Detection Benchmarks\n\nCode to reproduce results from the paper:\n\n[**Back to the Basics: Revisiting Out-of-Distribution Detection Baselines**](https://arxiv.org/abs/2207.03061). [*ICML 2022 Workshop on Principles of Distribution Shift*](https://icml.cc/Conferences/2022/ScheduleMultitrack?event=20541)\n\nOut-of-distribution (OOD) detection is the task of determining whether a datapoint comes from a different distribution than the training dataset. For example, we may train a model to classify the breed of dogs and find that there is a cat image in our dataset. This cat image would be considered out-of-distribution. \nThis work evaluates the effectiveness of various scores to detect OOD datapoints.\n\nThis repository is only for intended for scientific purposes. To detect outliers in your own data, you should instead use the [implementation](https://docs.cleanlab.ai/stable/tutorials/outliers.html) from the official [cleanlab](https://github.com/cleanlab/cleanlab) library.\n\n## File Structure\nThis repository is broken into two major folders (inside `src/experiments/`):\n\n1. `OOD/`: primary benchmarking code used for the paper linked above.\n\n2. `adjusted-OOD-scores/`: additional benchmarking code to produce results from the article:\n\n[**A Simple Adjustment Improves Out-of-Distribution Detection for Any Classifier**](https://pub.towardsai.net/a-simple-adjustment-improves-out-of-distribution-detection-for-any-classifier-5e96bbb2d627). *Towards AI*, 2022.\n\nThis additional code considers OOD detection based solely on classifier predictions and adjusted versions thereof. \n\n\n## Experiments\n\nFor each experiment, we perform the following procedure:\n\n1. Train a Neural Network model with ONLY the **in-distribution** training dataset.\n2. Use this model to generate predicted probabilties and embeddings for the **in-distribution** and **out-of-distribution** test datasets (these are considered out-of-sample predictions).\n3. Use out-of-sample predictions to generate OOD scores.\n4. Threshold OOD scores to detect OOD datapoints.\n\n| Experiment ID | In-Distribution | Out-of-Distribution |\n| :------------ | :-------------- | :------------------ |\n| 0             | cifar-10        | cifar-100           |\n| 1             | cifar-100       | cifar-10            |\n| 2             | mnist           | roman-numeral       |\n| 3             | roman-numeral   | mnist               |\n| 4             | mnist           | fashion-mnist       |\n| 5             | fashion-mnist   | mnist               |\n\n\n\n## Download datasets\n\nFor our experiments, we use [AutoGluon's ImagePredictor](https://auto.gluon.ai/dev/tutorials/image_prediction/beginner.html) for image classification which requires the training, validation, and test datasets to be image files.\n\nLinks below to download the training and test datasets in PNG format:\n\n- **cifar-10** and **cifar-100**:\n  https://github.com/knjcode/cifar2png\n\n- **roman-numeral**:\n  https://worksheets.codalab.org/bundles/0x497f5d7096724783aa1eb78b85aa321f\n\n  There are duplicate images in the dataset (exact same image with different file names). We use the following script to dedupe: `src/preprocess/remove_dupes.py`\n\n- **mnist**:\n  https://github.com/myleott/mnist_png\n\n- **fashion-mnist**:\n  https://github.com/DeepLenin/fashion-mnist_png\n\n\n## Instructions to reproduce results\n\n#### Prerequisite\n\n- [NVIDIA Container Toolkit](https://github.com/NVIDIA/nvidia-docker): allows us to properly utilize our NVIDIA GPUs inside docker environments\n- [autogluon](https://auto.gluon.ai/stable/index.html)==0.4.0\n\n\n#### 1. Run docker-compose to build the docker image and run the container\n\nClone this repo and run below commands:\n\n```bash\nsudo docker-compose build\nsudo docker-compose run --rm --service-port dcai\n```\n\n\n#### 2. Start Jupyter Lab\n\nRun command below.\n\nNote that we use a Makefile to run jupyter lab for convenience so we can save args (ip, port, allow-root, etc).\n\n```bash\nmake jupyter-lab\n```\n\n\n#### 3. Train models\n\nRun notebook below to train all models.\n\n[src/experiments/OOD/0_Train_Models.ipynb](https://github.com/JohnsonKuan/ood-detection-benchmarks/blob/main/src/experiments/OOD/0_Train_Models.ipynb)\n\nNote that we use 2 neural net architectures below with AutoGluon and each use different backends:\n\n- swin_base_patch4_window7_224 (torch backend)\n- resnet50_v1 (mxnet backend)\n\n\n#### 4. Run experiments\n\nHere is a notebook that runs all experiments:\n\n[src/experiments/OOD/1_Evaluate_All_OOD_Experiments.ipynb](https://github.com/JohnsonKuan/ood-detection-benchmarks/blob/main/src/experiments/OOD/1_Evaluate_All_OOD_Experiments.ipynb)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcleanlab%2Food-detection-benchmarks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcleanlab%2Food-detection-benchmarks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcleanlab%2Food-detection-benchmarks/lists"}