https://github.com/gpleiss/temperature_scaling
A simple way to calibrate your neural network.
https://github.com/gpleiss/temperature_scaling
calibration deep-learning
Last synced: 6 months ago
JSON representation
A simple way to calibrate your neural network.
- Host: GitHub
- URL: https://github.com/gpleiss/temperature_scaling
- Owner: gpleiss
- License: mit
- Created: 2017-08-03T13:32:44.000Z (about 8 years ago)
- Default Branch: master
- Last Pushed: 2021-08-24T11:37:06.000Z (about 4 years ago)
- Last Synced: 2025-04-04T01:08:13.825Z (6 months ago)
- Topics: calibration, deep-learning
- Language: Python
- Size: 18.6 KB
- Stars: 1,135
- Watchers: 10
- Forks: 163
- Open Issues: 26
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Temperature Scaling
A simple way to calibrate your neural network.
The `temperature_scaling.py` module can be easily used to calibrated any trained model.
Based on results from [On Calibration of Modern Neural Networks](https://arxiv.org/abs/1706.04599).
## Motivation
**TLDR:** Neural networks tend to output overconfident probabilities.
Temperature scaling is a post-processing method that fixes it.
**Long:**
Neural networks output "confidence" scores along with predictions in classification.
Ideally, these confidence scores should match the true correctness likelihood.
For example, if we assign 80% confidence to 100 predictions, then we'd expect that 80% of
the predictions are actually correct. If this is the case, we say the network is **calibrated**.
A simple way to visualize calibration is plotting accuracy as a function of confidence.
Since confidence should reflect accuracy, we'd like for the plot to be an identity function.
If accuracy falls below the main diagonal, then our network is overconfident.
This happens to be the case for most neural networks, such as this ResNet trained on CIFAR100.
Temperature scaling is a post-processing technique to make neural networks calibrated.
After temperature scaling, you can trust the probabilities output by a neural network:
Temperature scaling divides the logits (inputs to the softmax function) by a learned scalar parameter. I.e.
```
softmax = e^(z/T) / sum_i e^(z_i/T)
```
where `z` is the logit, and `T` is the learned parameter.
We learn this parameter on a validation set, where T is chosen to minimize NLL.
## Demo
First train a DenseNet on CIFAR100, and save the validation indices:
```sh
python train.py --data --save
```
Then temperature scale it
```sh
python demo.py --data --save
```
## To use in a project
Copy the file `temperature_scaling.py` to your repo.
Train a model, and **save the validation set**.
(You must use the same validation set for training as for temperature scaling).
You can do something like this:
```python
from temperature_scaling import ModelWithTemperature
orig_model = ... # create an uncalibrated model somehow
valid_loader = ... # Create a DataLoader from the SAME VALIDATION SET used to train orig_model
scaled_model = ModelWithTemperature(orig_model)
scaled_model.set_temperature(valid_loader)
```