Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/deeplearning-wisc/poem
PyTorch implementation of POEM (Out-of-distribution detection with posterior sampling), ICML 2022
https://github.com/deeplearning-wisc/poem
Last synced: about 2 months ago
JSON representation
PyTorch implementation of POEM (Out-of-distribution detection with posterior sampling), ICML 2022
- Host: GitHub
- URL: https://github.com/deeplearning-wisc/poem
- Owner: deeplearning-wisc
- Created: 2022-06-12T01:59:51.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-05-06T05:39:07.000Z (over 1 year ago)
- Last Synced: 2024-07-10T08:43:50.038Z (3 months ago)
- Language: Python
- Homepage:
- Size: 3 MB
- Stars: 28
- Watchers: 4
- Forks: 2
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Out-of-distribution Detection with Posterior Sampling
This project is for the paper: [POEM: Out-of-distribution Detection with Posterior Sampling](https://arxiv.org/abs/2206.13687) published at ICML 2022 (long talk). Some parts of the codebase are adapted from [ODIN](https://github.com/facebookresearch/odin), [Outlier Exposure](https://github.com/hendrycks/outlier-exposure), [Deep Mahalanobis Detector](https://github.com/pokaxpoka/deep_Mahalanobis_detector) and [NTOM](https://github.com/jfc43/informative-outlier-mining).
## Required Packages
The following packages are required to be installed:
- [PyTorch](https://pytorch.org/)
- [Scipy](https://github.com/scipy/scipy)
- [Numpy](http://www.numpy.org/)
- [Sklearn](https://scikit-learn.org/stable/)
Our experiments are conducted on Ubuntu Linux 20.04 with Python 3.8.
## Method Overview
## In-distribution and Auxiliary Outlier Datasets
- In-distribution training set:
- [CIFAR](https://www.cs.toronto.edu/~kriz/cifar.html): included in PyTorch.
* Auxiliary outlier training set:
* [80 Million Tiny Images](https://groups.csail.mit.edu/vision/TinyImages/): to download **80 Million Tiny Images** dataset. After downloading it, place it in this directory: `datasets/unlabeled_datasets/80M_Tiny_Images`
* ImageNet-RC (Downsampled ImageNet Datasets](https://patrykchrabaszcz.github.io/Imagenet32/): we use the ImageNet64x64, which can be downloaded from [ImageNet Website](http://image-net.org/download-images). After downloading it, place it in this directory: `datasets/unlabeled_datasets/ImageNet`.
## Out-of-distribution Test Datasets
We provide links and instructions to download each dataset:
* [SVHN](http://ufldl.stanford.edu/housenumbers/test_32x32.mat): download it and place it in the folder of `datasets/ood_datasets/svhn`. Then run `python select_svhn_data.py` to generate test subset.
* [Textures](https://www.robots.ox.ac.uk/~vgg/data/dtd/download/dtd-r1.0.1.tar.gz): download it and place it in the folder of `datasets/ood_datasets/dtd`.
* [Places365](http://data.csail.mit.edu/places/places365/test_256.tar): download it and place it in the folder of `datasets/ood_datasets/places365/test_subset`. We randomly sample 10,000 images from the original test dataset.
* [LSUN](https://www.dropbox.com/s/fhtsw1m3qxlwj6h/LSUN.tar.gz): download it and place it in the folder of `datasets/ood_datasets/LSUN`.
* [LSUN-resize](https://www.dropbox.com/s/moqh2wh8696c3yl/LSUN_resize.tar.gz): download it and place it in the folder of `datasets/ood_datasets/LSUN_resize`.
* [iSUN](https://www.dropbox.com/s/ssz7qxfqae0cca5/iSUN.tar.gz): download it and place it in the folder of `datasets/ood_datasets/iSUN`.
For example, run the following commands in the **root** directory to download **LSUN**:
```
cd datasets/ood_datasets
wget https://www.dropbox.com/s/fhtsw1m3qxlwj6h/LSUN.tar.gz
tar -xvzf LSUN.tar.gz
```
## Quick Start
For example, to test the model trained with POEM at 100 epochs on CIFAR-10, please download the corresponding checkpoint to `./checkpoints/CIFAR-10/POEM_Sample_1/checkpoint_100.pth.tar`. You can also train from scratch with `train_poem.py`. For each ID dataset, posterior sampling hyperparameters can be tuned for better performance especially the ones controlling the variances for noises and weights: `sigma` and `sigma_n`. We observe that training longer until the model fully converges is helpful. For more stable performance, we recommend training CIFAR-10 for 100 epochs and CIFAR-100 for 200 epochs. To reproduce the results in the paper, we provide checkpoints for tuned models [here](https://www.dropbox.com/sh/tsqwbob249gouzy/AABS21uLcB1TdwoHxMQpOXpia?dl=0).
To evaluate the OOD detection performance of POEM, for example, with a checkpoint named `POEM_Sample_1`:
```python
# Generate evaluation results for POEM:
python get_score.py --name POEM_Sample_1 --in-dataset CIFAR-10 --model-arch densenet --test_epochs 100
# Present OOD detection metrics based on results:
python get_results.py --name POEM_Sample_1 --in-dataset CIFAR-10 --test_epochs 100
```
### For bibtex citations
If you find our work helpful for your research, please consider citing our paper:
```
@inproceedings{ming2022poem,
author={Ming, Yifei and Fan, Ying and Li, Yixuan},
title={POEM: Out-of-distribution Detection with Posterior Sampling},
booktitle={International Conference on Machine Learning},
year={2022},
organization={PMLR}
}
```
### Issues and further discussions
Please create an issue for code-related questions. For clarification and further discussions, free feel to drop an email at [email protected].