{"id":13577487,"url":"https://github.com/dlmacedo/robust-deep-learning","last_synced_at":"2025-10-08T17:25:22.772Z","repository":{"id":58075795,"uuid":"529643647","full_name":"dlmacedo/robust-deep-learning","owner":"dlmacedo","description":"A project to train your model from scratch or fine-tune a pretrained model using the losses provided in this library to improve out-of-distribution detection and uncertainty estimation performances. Calibrate your model to produce enhanced uncertainty estimations. Detect out-of-distribution data using the defined score type and threshold.","archived":false,"fork":false,"pushed_at":"2022-11-26T12:57:37.000Z","size":4366,"stargazers_count":17,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-19T19:18:13.096Z","etag":null,"topics":["anomaly-detection","classification","deep-learning","deep-neural-networks","machine-learning","novelty-detection","ood-detection","open-set","open-set-recognition","out-of-distribution","out-of-distribution-detection","pytorch","robust-deep-learning","robust-machine-learning","trustworthy-ai","trustworthy-machine-learning","uncertainty-calibration","uncertainty-estimation","uncertainty-neural-networks"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dlmacedo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-08-27T16:50:44.000Z","updated_at":"2024-12-18T07:59:40.000Z","dependencies_parsed_at":"2023-01-23T16:01:12.283Z","dependency_job_id":null,"html_url":"https://github.com/dlmacedo/robust-deep-learning","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlmacedo%2Frobust-deep-learning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlmacedo%2Frobust-deep-learning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlmacedo%2Frobust-deep-learning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlmacedo%2Frobust-deep-learning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dlmacedo","download_url":"https://codeload.github.com/dlmacedo/robust-deep-learning/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247338461,"owners_count":20922985,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["anomaly-detection","classification","deep-learning","deep-neural-networks","machine-learning","novelty-detection","ood-detection","open-set","open-set-recognition","out-of-distribution","out-of-distribution-detection","pytorch","robust-deep-learning","robust-machine-learning","trustworthy-ai","trustworthy-machine-learning","uncertainty-calibration","uncertainty-estimation","uncertainty-neural-networks"],"created_at":"2024-08-01T15:01:21.881Z","updated_at":"2025-09-16T13:23:27.943Z","avatar_url":"https://github.com/dlmacedo.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"\u003cimg align=\"center\" src=\"assets/rdl.png\" width=\"750\"\u003e\n\n# The Robust Deep Learning Library\n\n\u003e\u003e **TRAIN OR FINE-TUNE**\n\nTrain your model from scratch or fine-tune a pretrained model using the losses provided in this library to improve out-of-distribution detection and uncertainty estimation performances.\n\n\u003e\u003e **CALIBRATE**\n\nCalibrate your model to produce enhanced uncertainty estimations.\n\n\u003e\u003e **DETECT**\n\nDetect out-of-distribution data using the defined score type and threshold. \n\n## Features\n\n- **Model Independent:**\n\nUse models from timm library or whatever you want.\n\n- **Data Independent:**\n\nMost cases work for any type of media (e.g., image, text, audio, and others).\n\n- **Large-Scale Models and Data:**\n\nTrain using large-scale models and data (e.g., ImageNet).\n\n- **Efficient Inferences:**\n\nThe trained models are as efficient as the ones trained using the cross-entropy loss. \n\n- **Hyperparameter-Free:**\n\nThere is no hyperparameter to tune. \"You only train once\" (YOTO).\n\n- **Standard Interface:**\n\nUse the same API to train models with improved robustness using different losses.\n\n- **No Need for Additional Data:**\n\nThe losses used in this library do not require collecting or using additional data. \n\n- **Temperature Calibration:**\n\nCalculate the Uncertainty Estimation and update the temperature of the output last layer. \n\n- **Scalability: More data, Bigger Models, Better Results!**\n\nEntropic losses perform better and better as the size of the data and model increase. \n\n- **Threshold Computation:**\n\nCompute the threshold for deciding regarding out-of-distribution examples. \n\n- **Scores Computation:**\n\nCompute the scores opting from a set of many different types available. \n\n- **Detect Out-of-Distribution:**\n\nDetect out-of-distribution examples using the computed scores. \n\n- **State-of-the-art:**\n\nSOTA results for out-of-distribution detection and uncertainty estimation.\n\n## Results\n\n### Dataset=ImageNet, Model=ResNet18, Near OOD=ImageNet-O\n\n| Loss [Score] | Class (ACC) | Near OOD (AUROC) |\n|:---|:---:|:---:|\n| Cross-Entropy [MPS] | 69.9 | 52.4 |\n| DisMax [MMLES] | 69.6 | 75.8 |\n\n### Dataset=CIFAR\n\n\u003cimg align=\"center\" src=\"assets/table.png\" width=\"750\"\u003e\n\n## Installation\n\n```bash\npip install robust-deep-learning\n```\n\n## Usage\n\n```python\n# Import the robust deep learning library as rdl\nimport robust_deep_learning as rdl\n\n#####################################################################################################\n#####################################################################################################\n# Training or Fine-tuning\n#####################################################################################################\n#####################################################################################################\n\n#########################################################\n# Option 1: Creating a custom model and defining the loss\n#########################################################\n\n# Create from a model definition file. \n# For example, import a class \"Model\" from a model definition file.\nmodel = Model()\n# Load a pretrained model if fine-tuning rather than training from scratch (random weights).\n# model.load_state_dict(torch.load(pre_trained_file_name, map_location=\"cuda:\" + str(args.gpu)))\n\n# Chance the output last layer of the model with the desired loss first part.\n# If the name of the output last layer of the model is unknown, print it with \"print(model)\".\n# For example, if the output last layer is called \"classifier\":\n# \"model.classifier = nn.Linear(num_features, num_classes)\"\n# Then, replace this output layer (usually a linear layer) by adding the following line: \nmodel.classifier = rdl.DisMaxLossFirstPart(num_features, num_classes)\n\n# Replace the Cross-Entropy Loss.\n# The first argument is the name of the output last layer of the model used above.\n# In case of training using a not too constrained model on image data, pass an add-on.\n# Currently, only DisMax has one add-on called \"FPR\".\n# Otherwise, do not pass add-on or simple pass \"add_on=None\". \ncriterion = rdl.DisMaxLossSecondPart(model.classifier, add_on=\"FPR\", gpu=None)\n\n#######################################################\n# Option 2: Creating a timm model and defining the loss\n#######################################################\n\n# For using a model from timm lib, use \"create_model\" functionality.\n# It is possible to start from a pretrained model to fine-tune using the desired loss.\nmodel = timm.create_model('resnet18', pretrained=False)\n\n# Chance the output last layer of the model with the desired loss first part.\n# If the name of the output last layer of the model is unknown, print it with \"print(model)\".\n# For example, if the output last layer is called \"fc\":\n# \"model.fc = nn.Linear(num_features, num_classes)\"\n# Then, replace this output layer (usually a linear layer) by adding the following line: \nmodel.fc = rdl.DisMaxLossFirstPart(model.get_classifier().in_features, num_classes)\n\n# Replace the Cross-Entropy Loss.\n# The first argument is the name of the output last layer of the model used above.\n# In case of training using a not too constrained model on image data, pass an add-on.\n# Currently, only DisMax has one add-on called \"FPR\".\n# Otherwise, do not pass add-on or simple pass \"add_on=None\". \ncriterion = rdl.DisMaxLossSecondPart(model.fc, add_on=\"FPR\", gpu=None)\n\n############################\n# Checking the training loop\n############################\n\nfor epoch in epochs:\n\n    # Training loop\n    for inputs, targets in in_data_train_loader:\n\n        # In the training loop, add the line below for preprocessing before forwarding.\n        # This is only required if using add_on other than None. Otherwise, this line is not needed.\n        inputs, targets = criterion.preprocess(inputs, targets) \n\n        # The three below lines are usually found in the training loop!\n        # These lines must not be changed!\n        outputs = model(inputs)\n        loss = criterion(outputs, targets)\n        loss.backward()\n\n#####################################################################################################\n#####################################################################################################\n# Uncertainty Estimation\n#####################################################################################################\n#####################################################################################################\n\n#############\n# Calibrating\n#############\n\n# Get outputs, labels, accuracy, expected calibration error (ECE), and negative log-likelihood (NLL).\nresults = rdl.get_outputs_labels_and_metrics(model, in_data_val_loader, gpu=None)\n\n# Calculate probabilities and verify the values returned above. \nprobabilities = torch.nn.Softmax(dim=1)(results[\"outputs\"])\nprint(probabilities)\nprint(results[\"acc\"], results[\"ece\"], results[\"nll\"])\n\n# Calibrate the temperature passing the output last layer, the model, and the validation set.\n# Choose the metric to optimize. For example, \"ECE\" (the only option for now).\nrdl.calibrate_temperature(model.classifier, model, in_data_val_loader, optimize=\"ECE\", gpu=None)\n#rdl.calibrate_temperature(model.fc, model, in_data_val_loader, optimize=\"ECE\")\n\n# Verify the novel value of the temperature of the output last layer.\nprint(model.classifier.temperature)\n\n######################\n# Verifing calibration\n######################\n\n# Get the results again using the calibrated model.\nresults = rdl.get_outputs_labels_and_metrics(model, in_data_val_loader, gpu=None)\n\n# Verifiy the probabilities, ECE, and NLL are improved regarding the now calibrated model.\nprobabilities = torch.nn.Softmax(dim=1)(results[\"outputs\"])\nprint(probabilities)\nprint(results[\"acc\"], results[\"ece\"], results[\"nll\"])\n\n####################################################################################################\n####################################################################################################\n# Out-of-Distribution Detection\n####################################################################################################\n####################################################################################################\n\n########################\n# Estimating performance\n########################\n\n# Define a score type. Typically, the best/recommended option for the loss you are using.\nscore_type = \"MMLES\"\n\n# Evaluate the out-of-distribution detection performance passing loaders.\nood_metrics = rdl.get_ood_metrics(\n    model, in_data_val_loader, out_data_loader, score_type, fpr=0.05, gpu=None)\n\n# Optionally, first get in-data scores:\nresults = rdl.get_outputs_labels_and_metrics(model, in_data_val_loader, gpu=None)\nin_data_scores = rdl.get_scores(results[\"outputs\"], score_type)\n\n# Second, get out-data scores:\nresults = rdl.get_outputs_labels_and_metrics(model, out_data_loader, gpu=None)\nout_data_scores = rdl.get_scores(results[\"outputs\"], score_type)\n\n# Then, finally, evaluate the out-of-distribution detection performance passing scores.\nood_metrics = rdl.get_ood_metrics_from_scores(in_data_scores, out_data_scores, fpr=0.05)\n\n######################################\n# Detecting (still testing this part))\n######################################\n\n# Before detecting, it is necessary to compute the thresholds.\nthresholds = rdl.get_thresholds(model, in_data_val_loader, score_type, gpu=None)\n\n# Optionaly, it is possible to compute the threshold using in-data scores.\n#thresholds = rdl.get_thresholds(results[\"outputs\"], score_types=\"MMLES\")\nthresholds = rdl.get_thresholds_from_scores(in_data_scores) # guarder \n\n# After calculating thresholds, detection may be performed.\n# Some test input may be obtained using: input = next(iter(in_data_val_loader))\nood_detections = rdl.get_ood_detections(model, inputs, thresholds, fpr=\"0.05\", gpu=None)\n```\n\n## Losses and Scores\n\nThe following losses are implemented:\n\n- Isotropy Maximization Loss [arXiv](https://arxiv.org/abs/2006.04005) [conference version](https://ieeexplore.ieee.org/document/9533899) [journal version](https://ieeexplore.ieee.org/document/9556483)\n- Enhanced Isotropy Maximization Loss [arXiv](https://arxiv.org/abs/2105.14399)\n- Distiction Maximization Loss [arXiv](https://arxiv.org/abs/2205.05874)\n\nThe following scores are implemented:\n\n- Maximum Probability Score\n- Entropic Score [arXiv](https://arxiv.org/abs/2006.04005) [conference version](https://ieeexplore.ieee.org/document/9533899) [journal version](https://ieeexplore.ieee.org/document/9556483)\n- Minimum Distance Score [arXiv](https://arxiv.org/abs/2105.14399)\n- Maximum Mean Logit Entropy Score [arXiv](https://arxiv.org/abs/2205.05874)\n\n## Experiments\n\nInstall the requirements to reproduce the experiments of this library:\n\n```bash\npip install -r requirements.txt\n```\n\nPlease, move to the `data` directory and run all the prepare data bash scripts:\n\n```bash\n# Download and prepare out-of-distrbution data for CIFAR10 and CIFAR100 datasets.\n./prepare-cifar.sh\n# Download and prepare out-of-distrbution data for ImageNet.\n./prepare-imagenet.sh\n```\n\nRun the experiments:\n\n```bash\n./run_cifar100_densenetbc100.sh*\n./run_cifar100_resnet34.sh*\n./run_cifar100_wideresnet2810.sh*\n./run_cifar10_densenetbc100.sh*\n./run_cifar10_resnet34.sh*\n./run_cifar10_wideresnet2810.sh*\n./run_imagenet1k_resnet18.sh*\n```\n\nAnalize the results:\n\n```bash\n./analize.sh\n```\n\n## Citation\n\nPlease, cite our papers if you use our losses in your work:\n\n```bibtex\n@INPROCEEDINGS{9533899,\n  author={Macêdo, David and Ren, Tsang Ing and Zanchettin, Cleber and Oliveira, \n  Adriano L. I. and Ludermir, Teresa},\n  booktitle={2021 International Joint Conference on Neural Networks (IJCNN)}, \n  title={Entropic Out-of-Distribution Detection}, \n  year={2021},\n  pages={1-8},\n  doi={10.1109/IJCNN52387.2021.9533899}\n}\n```\n\n```bibtex\n@ARTICLE{9556483,\n  author={Macêdo, David and Ren, Tsang Ing and Zanchettin, Cleber and Oliveira, \n  Adriano L. I. and Ludermir, Teresa},\n  journal={IEEE Transactions on Neural Networks and Learning Systems}, \n  title={Entropic Out-of-Distribution Detection:\n  Seamless Detection of Unknown Examples}, \n  year={2022},\n  volume={33},\n  number={6},\n  pages={2350-2364},\n  doi={10.1109/TNNLS.2021.3112897}\n}\n```\n\n```bibtex\n@article{DBLP:journals/corr/abs-2105-14399,\n  author    = {David Mac{\\^{e}}do and\n               Teresa Bernarda Ludermir},\n  title     = {Enhanced Isotropy Maximization Loss: \n  Seamless and High-Performance Out-of-Distribution Detection\n  Simply Replacing the SoftMax Loss},\n  journal   = {CoRR},\n  volume    = {abs/2105.14399},\n  year      = {2021},\n  url       = {https://arxiv.org/abs/2105.14399},\n  eprinttype = {arXiv},\n  eprint    = {2105.14399},\n  timestamp = {Wed, 02 Jun 2021 11:46:42 +0200},\n  biburl    = {https://dblp.org/rec/journals/corr/abs-2105-14399.bib},\n  bibsource = {dblp computer science bibliography, https://dblp.org}\n}\n```\n\n```bibtex\n@article{DBLP:journals/corr/abs-2205-05874,\n  author    = {David Mac{\\^{e}}do and\n               Cleber Zanchettin and\n               Teresa Bernarda Ludermir},\n  title     = {Distinction Maximization Loss:\n  Efficiently Improving Out-of-Distribution Detection and Uncertainty Estimation\n  Simply Replacing the Loss and Calibrating},\n  journal   = {CoRR},\n  volume    = {abs/2205.05874},\n  year      = {2022},\n  url       = {https://doi.org/10.48550/arXiv.2205.05874},\n  doi       = {10.48550/arXiv.2205.05874},\n  eprinttype = {arXiv},\n  eprint    = {2205.05874},\n  timestamp = {Tue, 17 May 2022 17:31:03 +0200},\n  biburl    = {https://dblp.org/rec/journals/corr/abs-2205-05874.bib},\n  bibsource = {dblp computer science bibliography, https://dblp.org}\n}\n```\n\n```bibtex\n@article{DBLP:journals/corr/abs-2208-03566,\n  author    = {David Mac{\\^{e}}do},\n  title     = {Towards Robust Deep Learning using Entropic Losses},\n  journal   = {CoRR},\n  volume    = {abs/2208.03566},\n  year      = {2022},\n  url       = {https://doi.org/10.48550/arXiv.2208.03566},\n  doi       = {10.48550/arXiv.2208.03566},\n  eprinttype = {arXiv},\n  eprint    = {2208.03566},\n  timestamp = {Wed, 10 Aug 2022 14:49:54 +0200},\n  biburl    = {https://dblp.org/rec/journals/corr/abs-2208-03566.bib},\n  bibsource = {dblp computer science bibliography, https://dblp.org}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdlmacedo%2Frobust-deep-learning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdlmacedo%2Frobust-deep-learning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdlmacedo%2Frobust-deep-learning/lists"}