{"id":13738270,"url":"https://github.com/MadryLab/backgrounds_challenge","last_synced_at":"2025-05-08T16:33:03.689Z","repository":{"id":45169977,"uuid":"272887265","full_name":"MadryLab/backgrounds_challenge","owner":"MadryLab","description":null,"archived":false,"fork":false,"pushed_at":"2020-10-02T00:15:53.000Z","size":891,"stargazers_count":138,"open_issues_count":4,"forks_count":16,"subscribers_count":9,"default_branch":"master","last_synced_at":"2024-11-15T07:34:04.090Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MadryLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-06-17T05:37:50.000Z","updated_at":"2024-10-10T11:49:39.000Z","dependencies_parsed_at":"2022-07-13T17:00:32.753Z","dependency_job_id":null,"html_url":"https://github.com/MadryLab/backgrounds_challenge","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MadryLab%2Fbackgrounds_challenge","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MadryLab%2Fbackgrounds_challenge/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MadryLab%2Fbackgrounds_challenge/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MadryLab%2Fbackgrounds_challenge/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MadryLab","download_url":"https://codeload.github.com/MadryLab/backgrounds_challenge/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253105412,"owners_count":21855019,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T03:02:16.657Z","updated_at":"2025-05-08T16:33:03.349Z","avatar_url":"https://github.com/MadryLab.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"The **backgrounds challenge** is a public dataset challenge for creating more background-robust models. This repository contains test datasets of ImageNet-9 (IN-9) with different amounts of background and foreground signal, which you can use to measure the extent to which your models rely on image backgrounds. These are described further in the paper: **\"Noise or Signal: The Role of Image Backgrounds in Object Recognition\"** ([preprint](https://arxiv.org/abs/2006.09994), [blog](http://gradsci.org/background)).\n\n# Backgrounds Challenge\nDeep computer vision models rely on both foreground objects and image backgrounds. Even when the correct foreground object is present, such models often make incorrect predictions when the image background is changed, and they are especially vulnerable to **adversarially chosen backgrounds**. For example, the [the official pre-trained PyTorch ResNet-50](https://pytorch.org/docs/stable/torchvision/models.html) has an accuracy of 22% when evaluated against adversarial backgrounds on ImageNet-9 (for reference, a model that always predicts \"dog\" has an accuracy of 11%).\n\nThus, the goal of this challenge is to understand how background-robust models can be. Specifically, we assess models by their accuracy on images containing foregrounds superimposed on backgrounds which are adversarially chosen from the test set. We encourage researchers to use this challenge to benchmark progress on background-robustness, which can be important for determining models' out of distribution performance. We will maintain a leaderboard of top submissions.\n\n\u003cimg align=\"center\" src=\"assets/adversarial_backgrounds_insect.png\" width=\"750\"\u003e\n\u003csub\u003e\u003csup\u003eExamples from the insect class of the most adversarial backgrounds for a model. The number above each image represents the proportion of non-insect foregrounds that can be fooled by these backgrounds.\u003c/sup\u003e\u003c/sub\u003e\n\n\n## Backgrounds Challenge Leaderboard\n\n| Model                     | Reference                     | Challenge \u003cbr\u003e Accuracy        | Clean Accuracy \u003cbr\u003e (on IN-9) | Download Link\n|---------------------------|-------------------------------|---------------------------|----------------|------------------------|\n| ResNet-50                 | (initial entry)               | 22.3%                     | 95.6%          | [Official Pytorch Model](https://download.pytorch.org/models/resnet50-19c8e357.pth)\n| ResNet-50 ([IN-9L](https://arxiv.org/abs/2006.09994))         | (initial entry)               | 12.0%                     | 96.4%          | [Download](https://www.dropbox.com/s/5bmfiunmlh0hy8n/in9l_resnet50.pt?dl=0)\n\n\n## Running the Backgrounds Challenge Evaluation\nTo evaluate your model against adversarial backgrounds, you will need to do the following:\n\n1. Download and unzip the datasets included in the [release](https://github.com/MadryLab/backgrounds_challenge/releases).\n2. Run `python challenge_eval.py --checkpoint '/PATH/TO/CHECKPOINT' --data-path '/PATH/TO/DATA'`.\n\nThe model checkpoint that the script takes as input must be one of the following.\n1. A 1000-class ImageNet classifier.\n2. A 9-class IN-9 classifier.\n\nSee `python challenge_eval.py -h` for how to toggle between the two.\n\n**Note**: evaluation requires PyTorch to be installed with CUDA support.\n\n## Submitting a Model\nWe invite any interested researchers to submit models and results by submitting a pull request with your model checkpoint included. The most successful models will be listed in the leaderboard above. We have already included baseline pre-trained models for reference.\n\n# Testing your model on ImageNet-9 and its variations\n\u003cimg align=\"center\" src=\"assets/imagenet9_insect.png\" width=\"750\"\u003e\n\u003csub\u003e\u003csup\u003eAll variations of IN-9; each variation contains different amounts of foreground and background signal.\u003c/sup\u003e\u003c/sub\u003e\n\u003cbr/\u003e\n\u003cbr/\u003e\n\nImageNet-9 and its variations can be useful for measuring the impact of backgrounds on model decision making\u0026mdash;see the [paper](https://arxiv.org/abs/2006.09994) for more details. You can test your own models on IN-9 and its variations as follows.\n\n1. Download and unzip the datasets included in the [release](https://github.com/MadryLab/backgrounds_challenge/releases).\n2. Run, for example, `python in9_eval.py --eval-dataset 'mixed_same' --checkpoint '/PATH/TO/CHECKPOINT' --data-path '/PATH/TO/DATA'`. You can replace `mixed_same` with whichever variation of IN-9 you are interested in.\n\nJust like in the challenge, the input can either be a 1000-class ImageNet model or a 9-class IN-9 model.\n\nThere is no leaderboard or challenge for these datasets, but we encourage researchers to use these datasets to measure the role of image background in their models' decision making. Furthermore, we include a table of results for common pre-trained models and various models discussed in the paper.\n\n## Test Accuracy Results on ImageNet-9\n| Model                      | Original        | Mixed-Same      | Mixed-Rand      | BG-Gap          |\n|----------------------------|-----------------|-----------------|-----------------|-----------------|\n| VGG16-BN                   | 94.3%           | 83.6%           | 73.4%           | 10.2%           |\n| ResNet-50                  | 95.6%           | 86.2%           | 78.9%           |  7.3%           |\n| ResNet-152                 | 96.7%           | 89.3%           | 83.5%           |  5.8%           |\n| ResNet-50 (IN-9L)          | 96.4%           | 89.8%           | 75.6%           | 14.2%           |\n| ResNet-50 (IN-9/Mixed-Rand)| 73.3%           | 71.5%           | 71.3%           |  0.2%           |\n\n\u003csub\u003e\u003csup\u003eThe BG-Gap, or the difference between Mixed-Same and Mixed-Rand, measures the impact of background correlations in the presence of correct-labeled foregrounds.\u003c/sup\u003e\u003c/sub\u003e\n\n## Training Data\n**Updated June 24, 2020**: We are releasing all training data that we used to train models described in the paper. The download links are as follows:\n[IN-9L](https://www.dropbox.com/s/8w29bg9niya19rn/in9l.tar.gz?dl=0),\n[Mixed-Next](https://www.dropbox.com/s/4hnkbvxastpcgz2/mixed_next.tar.gz?dl=0),\n[Mixed-Rand](https://www.dropbox.com/s/cto15ceadgraur2/mixed_rand.tar.gz?dl=0),\n[Mixed-Same](https://www.dropbox.com/s/f2525w5aqq67kk0/mixed_same.tar.gz?dl=0),\n[No-FG](https://www.dropbox.com/s/0v6w9k7q7i1ytvr/no_fg.tar.gz?dl=0),\n[Only-BG-B](https://www.dropbox.com/s/u1iekdnwail1d9u/only_bg_b.tar.gz?dl=0),\n[Only-BG-T](https://www.dropbox.com/s/03lk878q73hyjpi/only_bg_t.tar.gz?dl=0),\n[Only-FG](https://www.dropbox.com/s/alrf3jo8yyxzyrn/only_fg.tar.gz?dl=0),\n[Original](https://www.dropbox.com/s/0vv2qsc4ywb4z5v/original.tar.gz?dl=0).\n\nEach downloadable dataset contains both training data and validation data generated in the same way as the training data (that is, with no manual cleaning); this validation data can be safely ignored. The test data in the [release](https://github.com/MadryLab/backgrounds_challenge/releases) should be used instead.\n\n\n## Citation\n\nIf you find these datasets useful in your research, please consider citing:\n\n    @article{xiao2020noise,\n      title={Noise or Signal: The Role of Image Backgrounds in Object Recognition},\n      author={Kai Xiao and Logan Engstrom and Andrew Ilyas and Aleksander Madry},\n      journal={ArXiv preprint arXiv:2006.09994},\n      year={2020}\n    }\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMadryLab%2Fbackgrounds_challenge","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMadryLab%2Fbackgrounds_challenge","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMadryLab%2Fbackgrounds_challenge/lists"}