{"id":15627809,"url":"https://github.com/jacobgil/pytorch-explain-black-box","last_synced_at":"2025-04-07T12:08:51.264Z","repository":{"id":41207487,"uuid":"93309088","full_name":"jacobgil/pytorch-explain-black-box","owner":"jacobgil","description":"PyTorch implementation of Interpretable Explanations of Black Boxes by Meaningful Perturbation","archived":false,"fork":false,"pushed_at":"2021-11-30T15:29:56.000Z","size":579,"stargazers_count":335,"open_issues_count":5,"forks_count":51,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-03-31T10:11:12.668Z","etag":null,"topics":["blackbox","deep-learning","deeplearning","pytorch","visualization"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jacobgil.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-06-04T11:30:48.000Z","updated_at":"2025-03-21T20:26:22.000Z","dependencies_parsed_at":"2022-08-26T06:20:58.686Z","dependency_job_id":null,"html_url":"https://github.com/jacobgil/pytorch-explain-black-box","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jacobgil%2Fpytorch-explain-black-box","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jacobgil%2Fpytorch-explain-black-box/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jacobgil%2Fpytorch-explain-black-box/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jacobgil%2Fpytorch-explain-black-box/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jacobgil","download_url":"https://codeload.github.com/jacobgil/pytorch-explain-black-box/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247648978,"owners_count":20972945,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blackbox","deep-learning","deeplearning","pytorch","visualization"],"created_at":"2024-10-03T10:19:43.852Z","updated_at":"2025-04-07T12:08:51.244Z","avatar_url":"https://github.com/jacobgil.png","language":"Python","funding_links":[],"categories":["Paper implementations｜论文实现","Paper implementations"],"sub_categories":["Other libraries｜其他库:","Other libraries:"],"readme":"## PyTorch implementation of Interpretable Explanations of Black Boxes by Meaningful Perturbation ##\nThe paper: https://arxiv.org/abs/1704.03296\n#### What makes the deep learning network think the image label is 'pug, pug-dog' and 'tabby, tabby cat':\n![Dog](https://github.com/jacobgil/pytorch-explain-black-box/blob/master/examples/dog.png?raw=true) ![Cat](https://github.com/jacobgil/pytorch-explain-black-box/blob/master/examples/cat.png?raw=true)\n\n#### A perturbation of the dog that caused the dog category score to vanish:\n![Perturbed](https://github.com/jacobgil/pytorch-explain-black-box/blob/master/examples/perturbated_dog.png?raw=true)\n\n\n### What makes the deep learning network think the image label is 'flute, transverse flute':\n![Flute](https://github.com/jacobgil/pytorch-explain-black-box/blob/master/examples/flute_cam.jpg?raw=true)\n\n\n----------\nUsage: `python explain.py \u003cpath_to_image\u003e`\n\nThis is a PyTorch impelentation of \n\n***\"Interpretable Explanations of Black Boxes by Meaningful Perturbation. Ruth Fong, Andrea Vedaldi\"***  with some deviations.\n\nThis uses VGG19 from torchvision. It will be downloaded when used for the first time.\n\nThis learns a mask of pixels that explain the result of a black box.\nThe mask is learned by posing an optimization problem and solving directly for the mask values.\n\nThis is different than other visualization techniques like Grad-CAM that use heuristics like high positive gradient values as an indication of relevance to the network score.\n\n\nIn our case the black box is the VGG19 model, but this can use any differentiable model.\n\n----------\n# How it works\n![Equation](https://github.com/jacobgil/pytorch-explain-black-box/blob/master/examples/equation.png?raw=true)\n\n*Taken from the paper https://arxiv.org/abs/1704.03296*\n\nThe goal is to solve for a mask that explains why did the network output a score for a certain category.\n\nWe create a low resolution (28x28) mask, and use it to perturb the input image to a deep learning network.\n\nThe perturbation combines a blurred version of the image, the regular image, and the up-sampled mask.\n\nWherever the mask contains low values, the input image will become more blurry.\n\nWe want to optimize for the next properties:\n\n 1. When using the mask to blend the input image and it's blurred versions, the score of the target category should drop significantly. \nThe evidence of the category should be removed!\n 2. The mask should be sparse. Ideally the mask should be the minimal possible mask to drop the category score.  This translates to a L1(1 - mask) term in the cost function.\n 3. The mask should be smooth.\n This translates to a total variation regularization in the cost function.\n 4. The mask shouldn't over-fit the network. Since the network activations might contain a lot of noise, it can be easy for the mask to just learn random values that cause the score to drop without being visually coherent.\n In addition to the other terms, this translates to solving for a lower resolution 28x28 mask.\n\n\n----------\n\n# Deviations from the paper\nThe paper uses a gaussian kernel with a sigma that is modulated by the value of the mask.\nThis is computational costly  to compute since the mask values are updated during the iterations, \nmeaning we need a different kernel for every mask pixel for every iteration.\n\nInitially I tried approximating this by first filtering the image with a filter bank of varying gaussian kernels. \nThen during optimization, the input image pixel would use the quantized mask value to select an appropriate filter bank output pixel (high mask value -\u003e lower channel).\n\nThis was done using the PyTorch variable gather/select_index functions.\nBut it turns out that the gather and select_index functions in PyTorch are not differentiable by the indexes.\n\n\nInstead, we just compute a perturbed image once, and then blend the image and the perturbed image using:\n\n`input_image = (1 - mask) * image + mask * perturbed_image`\n\nAnd it works well in practice.\n\nThe perturbed image here is the average of the gaussian and median blurred image, \nbut this can really be changed to many other combinations (try it out and find something better!).\n\n\nAlso now gaussian noise with a sigma of 0.2 is added to the preprocssed image at each iteration,\ninspired by google's SmoothGradient.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjacobgil%2Fpytorch-explain-black-box","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjacobgil%2Fpytorch-explain-black-box","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjacobgil%2Fpytorch-explain-black-box/lists"}