{"id":13578919,"url":"https://github.com/gpleiss/temperature_scaling","last_synced_at":"2025-04-12T21:28:36.084Z","repository":{"id":47263821,"uuid":"99235882","full_name":"gpleiss/temperature_scaling","owner":"gpleiss","description":"A simple way to calibrate your neural network.","archived":false,"fork":false,"pushed_at":"2021-08-24T11:37:06.000Z","size":19,"stargazers_count":1135,"open_issues_count":26,"forks_count":163,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-04-04T01:08:13.825Z","etag":null,"topics":["calibration","deep-learning"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gpleiss.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-08-03T13:32:44.000Z","updated_at":"2025-04-02T02:41:10.000Z","dependencies_parsed_at":"2022-08-12T13:50:09.763Z","dependency_job_id":null,"html_url":"https://github.com/gpleiss/temperature_scaling","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpleiss%2Ftemperature_scaling","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpleiss%2Ftemperature_scaling/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpleiss%2Ftemperature_scaling/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpleiss%2Ftemperature_scaling/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gpleiss","download_url":"https://codeload.github.com/gpleiss/temperature_scaling/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248633993,"owners_count":21136960,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["calibration","deep-learning"],"created_at":"2024-08-01T15:01:35.053Z","updated_at":"2025-04-12T21:28:36.061Z","avatar_url":"https://github.com/gpleiss.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Temperature Scaling\nA simple way to calibrate your neural network.\nThe `temperature_scaling.py` module can be easily used to calibrated any trained model.\n\nBased on results from [On Calibration of Modern Neural Networks](https://arxiv.org/abs/1706.04599).\n\n## Motivation\n\n**TLDR:** Neural networks tend to output overconfident probabilities.\nTemperature scaling is a post-processing method that fixes it.\n\n**Long:**\n\nNeural networks output \"confidence\" scores along with predictions in classification.\nIdeally, these confidence scores should match the true correctness likelihood.\nFor example, if we assign 80% confidence to 100 predictions, then we'd expect that 80% of\nthe predictions are actually correct. If this is the case, we say the network is **calibrated**.\n\nA simple way to visualize calibration is plotting accuracy as a function of confidence.\nSince confidence should reflect accuracy, we'd like for the plot to be an identity function.\nIf accuracy falls below the main diagonal, then our network is overconfident.\nThis happens to be the case for most neural networks, such as this ResNet trained on CIFAR100.\n\n![Uncalibrated ResNet](https://user-images.githubusercontent.com/824157/28974416-51ba7be4-7904-11e7-89ff-3c9b0ec4b607.png)\n\nTemperature scaling is a post-processing technique to make neural networks calibrated.\nAfter temperature scaling, you can trust the probabilities output by a neural network:\n\n![Calibrated ResNet](https://user-images.githubusercontent.com/824157/28974415-51ae78a8-7904-11e7-9b33-8fbe1f7c0a53.png)\n\nTemperature scaling divides the logits (inputs to the softmax function) by a learned scalar parameter. I.e.\n```\nsoftmax = e^(z/T) / sum_i e^(z_i/T)\n```\nwhere `z` is the logit, and `T` is the learned parameter.\nWe learn this parameter on a validation set, where T is chosen to minimize NLL.\n\n\n## Demo\n\nFirst train a DenseNet on CIFAR100, and save the validation indices:\n```sh\npython train.py --data \u003cpath_to_data\u003e --save \u003csave_folder_dest\u003e\n```\n\nThen temperature scale it\n```sh\npython demo.py --data \u003cpath_to_data\u003e --save \u003csave_folder_dest\u003e\n```\n\n\n## To use in a project\n\nCopy the file `temperature_scaling.py` to your repo.\nTrain a model, and **save the validation set**.\n(You must use the same validation set for training as for temperature scaling).\nYou can do something like this:\n\n```python\nfrom temperature_scaling import ModelWithTemperature\n\norig_model = ... # create an uncalibrated model somehow\nvalid_loader = ... # Create a DataLoader from the SAME VALIDATION SET used to train orig_model\n\nscaled_model = ModelWithTemperature(orig_model)\nscaled_model.set_temperature(valid_loader)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgpleiss%2Ftemperature_scaling","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgpleiss%2Ftemperature_scaling","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgpleiss%2Ftemperature_scaling/lists"}