{"id":13906795,"url":"https://github.com/suinleelab/attributionpriors","last_synced_at":"2025-12-26T01:42:34.619Z","repository":{"id":57412472,"uuid":"193598358","full_name":"suinleelab/attributionpriors","owner":"suinleelab","description":"Tools for training explainable models using attribution priors.","archived":false,"fork":false,"pushed_at":"2021-03-19T19:43:58.000Z","size":100021,"stargazers_count":121,"open_issues_count":2,"forks_count":9,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-11-16T03:37:14.026Z","etag":null,"topics":["artificial-intelligence","interpretable-deep-learning","machine-learning","tensorflow"],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/suinleelab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-06-24T23:54:24.000Z","updated_at":"2024-10-31T05:27:37.000Z","dependencies_parsed_at":"2022-09-10T03:11:32.443Z","dependency_job_id":null,"html_url":"https://github.com/suinleelab/attributionpriors","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/suinleelab%2Fattributionpriors","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/suinleelab%2Fattributionpriors/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/suinleelab%2Fattributionpriors/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/suinleelab%2Fattributionpriors/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/suinleelab","download_url":"https://codeload.github.com/suinleelab/attributionpriors/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226353472,"owners_count":17611708,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","interpretable-deep-learning","machine-learning","tensorflow"],"created_at":"2024-08-06T23:01:42.665Z","updated_at":"2025-12-26T01:42:34.611Z","avatar_url":"https://github.com/suinleelab.png","language":"Jupyter Notebook","funding_links":[],"categories":["模型的可解释性"],"sub_categories":[],"readme":"# Attribution Priors\n\nA repository for training explainable models using attribution priors.\n\nThis repository contains tools for connecting the machine learning topics of *model priors* \nand *model explanations* with a new method called *attribution priors*, discussed in our [paper](https://arxiv.org/abs/1906.10670) \n\"Learning Explainable Models Using Attribution Priors\". This package contains: \n* A differentiable axiomatic feature attribution method called *expected gradients*.\n* Tensorflow and PyTorch operations to directly regularize expected gradients attributions during training. \n* Examples of how arbitrary differentiable functions of expected gradient attributions can be regularized during training to encode prior knowledge about a modeling task. \n\nFor more guidance about how to use this repository/how to train with attribution priors, one of the quickest demos for PyTorch is available in the\n`Convergence Demo.ipynb` notebook. For quick demos in Tensorflow, see the `example_usage.ipynb` notebook \n(for older versions of TensorFlow) and the `example_usage_tf2.ipynb` notebook (for TensorFlow 2.0 and above)\nin the top level directory of this repository, and the [Installation and Usage Section](#installation) of this README. \n\n## Compatability\n\nThe code in this repository was written to support [TensorFlow versions](https://www.tensorflow.org/install) r1.8 and up, and works with both Python 2 and 3. If you are using TensorFlow with eager execution/TensorFlow 2.0 and above, see [Training with Eager Execution](#usage-training-with-eager-execution). If you are training with TensorFlow Sessions (old-school TensorFlow), see [Training with TensorFlow Sessions](#usage-training-with-tensorflow-sessions). We also now have support for [PyTorch](https://pytorch.org/get-started/locally/), which has been tested with Python 3 - see [Training with PyTorch](#usage-training-with-pytorch).\n\nCode has been tested for GPU compatability on a Lambda Blade GPU Server running CentOS 7.8, and for CPU compatibility on a MacBook Pro running macOS Catalina 10.15.4.\n\n## Installation\nThe easiest way to install this package is by cloning the repository:\n```\ngit clone https://github.com/suinleelab/attributionpriors.git\n```\nInstallation should take less than a minute.\n## Demo\nThe quickest demo to run the code is the `Convergence Demo.ipynb` notebook, available in the main directory. This notebook should run in a matter of minutes using only cpu, and also illustrates the benefits of expected gradients feature attributions over other feature attribution methods for attribution priors. For quick demos in Tensorflow, see the `example_usage.ipynb` notebook \n(for older versions of TensorFlow) and the `example_usage_tf2.ipynb` notebook (for TensorFlow 2.0 and above)\nin the top level directory of this repository, and the [Installation and Usage Section](#installation) of this README. \n\n## Examples \n\nSo what exactly are attribution priors and why would you want to use them? The examples here provide three ways in which\nyou can use attribution priors to improve network performance and interpretability. We use these examples\nin our paper. However, attribution priors are not limited to the examples here.\n\n### Image Data (mnist)\nIn the `mnist` folder, we give examples about how to train models that have smoother attributions over pixels, which in turn\nleads to better performance on noisy test data. Click through the notebooks in that folder to see more.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/suinleelab/attributionpriors/master/images/image.png\" width=\"1000\" /\u003e\n\u003c/p\u003e\n\n### Gene Expression Data (graph)\nIn the `graph` folder, the notebook shows how penalizing differences between the attributions of neighbors in an arbitrary graph \nconnecting the features can be used to incorporate prior biological knowledge about the relationships between genes, \nyield more biologically plausible explanations of drug response predictions, and improve test error.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/suinleelab/attributionpriors/master/images/graph.png\" width=\"1000\" /\u003e\n\u003c/p\u003e\n\n### Tabular Data (sparsity)\nIn the `sparsity` folder, the notebook shows how encouraging inequality in the distribution of feature attributions \ncan build sparser models that can perform more accurately when training data is limited. \n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/suinleelab/attributionpriors/master/images/sparsity.png\" width=\"1000\" /\u003e\n\u003c/p\u003e\n\n## Usage: Training with Eager Execution\n\nThis code provides an API for users who are using TensorFlow with eager execution, which is the default in TensorFlow 2.0 and above. The API change is rather simple in eager exceution and follows the following steps:\n\n### 1: Importing\n```python\n#Other import statements...\nfrom attributionpriors import eager_ops\n```\n\n### 2: Manually writing the train_step\nWhere normally you would write code like this:\n```python\n@tf.function\ndef train_step(inputs, labels, model):\n    with tf.GradientTape() as tape:\n        tape.watch(inputs)\n        predictions = model(inputs, training=True)\n        pred_loss = loss_fn(labels, predictions)\n        total_loss = pred_loss\n\n        if len(model.losses) \u003e 0:\n            regularization_loss = tf.math.add_n(model.losses)\n            total_loss = total_loss + regularization_loss\n\n    gradients = tape.gradient(total_loss, model.trainable_variables)\n    optimizer.apply_gradients(zip(gradients, model.trainable_variables))\n```\n\nNow you should add the following lines:\n```diff\n@tf.function\ndef train_step(inputs, labels, model):\n    with tf.GradientTape() as tape:\n        tape.watch(inputs)\n        predictions = model(inputs, training=True)\n        pred_loss = loss_fn(labels, predictions)\n        total_loss = pred_loss\n\n        if len(model.losses) \u003e 0:\n            regularization_loss = tf.math.add_n(model.losses)\n            total_loss = total_loss + regularization_loss\n\n+        attributions = eager_ops.expected_gradients(inputs, labels, model)\n+        attribution_loss = ap_loss_func(attributions, model)\n+        total_loss = total_loss + lamb * attribution_loss\n        \n    gradients = tape.gradient(total_loss, model.trainable_variables)\n    optimizer.apply_gradients(zip(gradients, model.trainable_variables))\n```\n\nwhere `ap_loss_func` is some loss function on top of your attributions, and `lamb` is a scalar\npenalty controlling the trade-off between penalizing attributions and your standard training loss. \nIn the image case, we use `tf.reduce_mean(tf.image.total_variation())`, the mean of the total variation across the attributions. \nAnd that's it! If you want a more in-depth example, see the `example_usage_tf2.ipynb` notebook.\n\n### Why can't we use the fit function?\nOverloading the fit function is, well, more difficult and makes it harder to specify complex penalties on your attributions. \nIt would require sub-classing the tf.keras.model API, which requires handling a variety of edge cases. If you want to \ntake on this project, feel free to do so, but we don't have plans to support it as of right now.\n\n\n## Usage: Training with TensorFlow Sessions\n\nThis package provides a simple API that can be used to define attribution priors over neural networks that can be dropped in to existing TensorFlow code.\nIn order to train using our implementation of attribution priors in your own code, you need to follow the following four steps:\n\n### 1: Importing\n```python\n#Other import statements...\nfrom attributionpriors.ops import AttributionPriorExplainer\n```\n### 2: Wrapping your input tensor using an AttributionPriorExplainer object\nWhere normally you would write code like this:\n```python\ninput_op = get_input() #Returns some tensor of shape [batch_size, feature_dim0, feature_dim1, ...] that you feed into a TensorFlow model\n```\nAdd the following lines in green:\n```diff\n+ explainer = AttributionPriorExplainer()\n  input_op = get_input()\n+ input_op, train_eg = explainer.input_to_samples_delta(input_op)\n```\n\n`input_op` can be a placeholder, a constant, or any other TensorFlow tensor that you use to feed data, like images or feature vectors, into a model. \nThe above code modifications create operations that alternate between feeding your model normal input and input interpolated between a sample and a reference.\nThis is used to compute expected gradients.\n\n### 3: Defining the Expected Gradients Tensor\nThe code for getting your expected gradients tensor depends on whether your model outputs a single number (e.g. in regression), \nor multiple (e.g. in classification tasks).\n#### Single-output Tasks (like regression)\nIf your code to define your model normally looks like this:\n```python\noutput_op = model(input_op) #output_op is a [batch_size] shaped tensor of floats\n```\nAdd the following line in green:\n```diff\n  output_op = model(input_op)\n+ expected_grads_op = explainer.shap_value_op(y_pred, cond_input_op)\n```\n\n#### Multi-output Tasks (like classification)\nIf instead, you are predicting multiple classes, you need to define which class you want to take expected gradients with respect to.\nAlthough our repository supports taking expected gradients with respect to ALL output classes (simply by using the code above in the single-output tasks), \nit is more computationally efficient and more intuitive to take expected gradients with respect to the _true class_. That is, if you have an image of, say, a car,\nyou should take attributions with respect to the car output class. Our repository supports this as follows:\n```diff\nlabel_op  = get_labels() #label_op should be a [batch_size] shaped tensor of integers specifying which class each input in input_op belongs to\noutput_op = model(input_op) #output_op is a [batch_size] shaped tensor of floats\n+ expected_grads_op = explainer.shap_value_op(y_pred, cond_input_op, label_op)\n```\n\nThis will return a tensor operation that represents attributions with respect to the true class of each example.\n\n### 4: Training with an Attribution Prior\nFirst, you need to define a loss function with respect to the `expected_grads_op` tensor. You can look to the examples section of the README for some examples\nof attribution priors, or you can read our original paper. For example, we use the total variation loss for images to get models with smoother explanations:\n```python\neg_loss_op  = tf.reduce_mean(tf.image.total_variation(expected_grads_op))\neg_train_op = tf.train.AdamOptimizer(learning_rate).minimize(eg_loss_op)\n```\n\nOnce you have an operation that trains your eg loss, you can call it alternating with your normal loss. If your code normally looks like this:\n```python\nfor i in range(num_training_steps):\n    sess.run(train_op)\n```\nInstead you should write:\n```diff\nfor i in range(num_training_steps):\n-    sess.run(train_op)\n+    batch_input, batch_labels, _ = sess.run([input_op, label_op, train_op])\n+    sess.run(eg_train_op, feed_dict={train_eg: True,\n+                                     input_op: batch_input,\n+                                     label_op: batch_labels})\n```\n\nIt is important that you set `train_eg: True` when you are running any operation related to the expected gradients tensor. \nIf you do not do so, the code will give strange errors. \n\nAnd that is all there is to it! Your code will simultaneously minimize\nboth your task objective and also whatever attribution prior you defined.\n\n## Usage: Training with PyTorch\n\nThis code provides an API for users who are using PyTorch to train their models.\n\n### 1: Importing\n```python\n#Other import statements...\nfrom attributionpriors.pytorch_ops import AttributionPriorExplainer\n```\n\n### 2: Initializing AttributionPriorExplainer\nBefore training, initialize the AttributionPriorExplainer object with the PyTorch Dataset object you want to use as background (we recommend using the full training dataset), the batch size, and the k parameter (number of background references per foreground samples).\n```python\nAPExp = AttributionPriorExplainer(background_dataset, batch_size,k=1)\n```\n\n### 3: Adding Expected Gradients Calculation to Training Step\nWhere your normal training loop in PyTorch might look like the following...\n```python\nfor features, labels in train_loader:\n    features, labels = features.cuda().float(), labels.cuda().float()\n    optimizer.zero_grad()\n        \n    outputs = model(features)\n    \n    loss = torch.nn.MSELoss(outputs, labels)\n    \n    loss.backward(retain_graph=True)\n    optimizer.step()\n    train_losses.append(loss.item())\n```\nNow simply add the following lines to, for example, add an L1 penalty on the expected gradients...\n```diff\nfor features, labels in train_loader:\n    features, labels = features.cuda().float(), labels.cuda().float()\n    optimizer.zero_grad()\n        \n    outputs = model(features)\n    \n-   loss = torch.nn.MSELoss(outputs, labels)\n+   expected_gradients = APExp.shap_values(model,features)\n+   attribution_prior = torch.norm(expected_gradients, p=1)\n+   loss = torch.nn.MSELoss(outputs, labels) + attribution_prior\n    \n    loss.backward(retain_graph=True)\n    optimizer.step()\n    train_losses.append(loss.item())\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsuinleelab%2Fattributionpriors","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsuinleelab%2Fattributionpriors","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsuinleelab%2Fattributionpriors/lists"}