{"id":22848345,"url":"https://github.com/feedzai/timeshap","last_synced_at":"2025-04-05T22:04:25.798Z","repository":{"id":37837138,"uuid":"446530335","full_name":"feedzai/timeshap","owner":"feedzai","description":"TimeSHAP explains Recurrent Neural Network predictions.","archived":false,"fork":false,"pushed_at":"2023-12-21T12:00:39.000Z","size":1600,"stargazers_count":171,"open_issues_count":12,"forks_count":32,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-03-29T21:03:59.010Z","etag":null,"topics":["explainability","rnn","shap","shapley-values"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/feedzai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-10T18:03:06.000Z","updated_at":"2025-03-25T09:22:13.000Z","dependencies_parsed_at":"2023-12-21T13:55:04.066Z","dependency_job_id":"c64af91f-74a5-437f-9e7f-1ddf776eac9b","html_url":"https://github.com/feedzai/timeshap","commit_stats":{"total_commits":50,"total_committers":5,"mean_commits":10.0,"dds":0.36,"last_synced_commit":"218f2ba385cb72f73ad245d249996f8566e65360"},"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feedzai%2Ftimeshap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feedzai%2Ftimeshap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feedzai%2Ftimeshap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feedzai%2Ftimeshap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/feedzai","download_url":"https://codeload.github.com/feedzai/timeshap/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247406085,"owners_count":20933803,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["explainability","rnn","shap","shapley-values"],"created_at":"2024-12-13T04:11:28.998Z","updated_at":"2025-04-05T22:04:25.768Z","avatar_url":"https://github.com/feedzai.png","language":"Jupyter Notebook","readme":"# TimeSHAP\n\n![PyPI version](https://badgen.net/pypi/v/timeshap)\n[![Downloads](https://static.pepy.tech/badge/timeshap)](https://pepy.tech/project/timeshap)\n\u003c!-- ![Python compatibility](https://badgen.net/pypi/python/timeshap) --\u003e\n\n\nTimeSHAP is a model-agnostic, recurrent explainer that builds upon KernelSHAP and \nextends it to the sequential domain.\nTimeSHAP computes event/timestamp- feature-, and cell-level attributions. \nAs sequences can be arbitrarily long, TimeSHAP also implements a pruning algorithm\nbased on Shapley Values, that finds a subset of consecutive, recent events that contribute\nthe most to the decision.\n\n\nThis repository is the code implementation of the TimeSHAP algorithm \npresent in the paper `TimeSHAP: Explaining Recurrent Models through Sequence Perturbations`\npublished at **KDD 2021**. \n\nLinks to the paper [here](https://arxiv.org/abs/2012.00073), \nand to the video presentation [here](https://www.youtube.com/watch?v=Q7Q9o7ywXx8).\n\n\n## Install TimeSHAP\n\n##### Via Pip\n```\npip install timeshap\n```\n\n##### Via Github\nClone the repository into a local directory using:\n```\ngit clone https://github.com/feedzai/timeshap.git\n```\n\nMove into the cloned repo and install the package:\n\n```\ncd timeshap\npip install .\n```\n\n\n##### Test your installation\nStart a Python session in your terminal using\n\n```\npython\n```\n\nAnd import TimeSHAP\n\n```\nimport timeshap\n```\n\n## TimeSHAP in 30 seconds\n\n#### Inputs\n- Model being explained;\n- Instance(s) to explain;\n- Background instance.\n\n#### Outputs\n- Local pruning output; (explaining a single instance)\n- Local event explanations; (explaining a single instance)\n- Local feature explanations; (explaining a single instance)\n- Global pruning statistics; (explaining multiple instances)\n- Global event explanations; (explaining multiple instances)\n- Global feature explanations; (explaining multiple instances)\n\n### Model Interface\n\nIn order for TimeSHAP to explain a model, an entry point must be provided.\nThis `Callable` entry point must receive a 3-D numpy array, `(#sequences; #sequence length; #features)`\nand return a 2-D numpy array `(#sequences; 1)` with the corresponding score of each sequence.\n\nIn addition, to make TimeSHAP more optimized, it is possible to return the **hidden state**\nof the model together with the score (if applicable). Although this is optional, we highly recommended it, \nas it has a very high impact. \nIf you choose to return the hidden state, this hidden state should either be: \n(see [notebook](notebooks/AReM/AReM_API_showcase.ipynb) for specific examples)\n - a 3-D numpy array, `(#rnn layers, #sequences, #hidden_dimension)` (class `ExplainedRNN` on notebook);\n - a tuple of numpy arrays that follows the previously described characteristic \n (usually used when using stacked RNNs with different hidden dimensions) (class `ExplainedGRU2Layer` on notebook); \n - a tuple of tuples of numpy arrays (usually used when using LSTM's) (class `ExplainedLSTM` on notebook);;\nTimeSHAP is able to explain any black-box model as long as it complies with the \npreviously described interface, including both PyTorch and TensorFlow models, \nboth examplified in our tutorials ([PyTorch](notebooks/AReM/AReM.ipynb), [TensorFlow](notebooks/AReM/AReM_TF.ipynb)).\n\nExample provided in our tutorials:\n- **TensorFLow**\n```\nmodel = tf.keras.models.Model(inputs=inputs, outputs=ff2)\nf = lambda x: model.predict(x)\n```\n\n- **Pytorch** - (Example where model receives and returns hidden states)\n```\nmodel_wrapped = TorchModelWrapper(model)\nf_hs = lambda x, y=None: model_wrapped.predict_last_hs(x, y)\n```\n\n\n###### Model Wrappers\nIn order to facilitate the interface between models and TimeSHAP, \nTimeSHAP implements `ModelWrappers`. These wrappers, used on the PyTorch\n[tutorial](notebooks/AReM/AReM.ipynb) notebook, allow for greater flexibility\nof explained models as they allow:\n- **Batching logic**: useful when using very large inputs or NSamples, which cannot fit\non GPU memory, and therefore batching mechanisms are required;\n- **Input format/type**: useful when your model does not work with numpy arrays. This\nis the case of our provided PyToch example; \n- **Hidden state logic**: useful when the hidden states of your models do not match\nthe hidden state format required by TimeSHAP\n\n\n### TimeSHAP Explanation Methods\nTimeSHAP offers several methods to use depending on the desired explanations.\nLocal methods provide detailed view of a model decision corresponding\nto a specific sequence being explained.\nGlobal methods aggregate local explanations of a given dataset\nto present a global view of the model.\n\n#### Local Explanations\n##### Pruning\n\n[`local_pruning()`](src/timeshap/explainer/pruning.py) performs the pruning\nalgorithm on a given sequence with a given user defined tolerance and returns \nthe pruning index along the information for plotting.\n\n[`plot_temp_coalition_pruning()`](src/timeshap/plot/pruning.py) plots the pruning \nalgorithm information calculated by `local_pruning()`.\n\n\u003cimg src=\"resources/images/pruning.png\" width=\"400\"\u003e\n\n##### Event level explanations\n\n[`local_event()`](src/timeshap/explainer/event_level.py) calculates event level explanations\nof a given sequence with the user-given parameteres and returns the respective \nevent-level explanations.\n\n[`plot_event_heatmap()`](src/timeshap/plot/event_level.py) plots the event-level explanations\ncalculated by `local_event()`.\n\n\u003cimg src=\"resources/images/event_level.png\" width=\"275\"\u003e\n\n##### Feature level explanations\n\n[`local_feat()`](src/timeshap/explainer/feature_level.py) calculates feature level explanations\nof a given sequence with the user-given parameteres and returns the respective \nfeature-level explanations.\n\n[`plot_feat_barplot()`](src/timeshap/plot/feature_level.py) plots the feature-level explanations\ncalculated by `local_feat()`.\n\n\u003cimg src=\"resources/images/feature_level.png\" width=\"350\"\u003e\n\n##### Cell level explanations\n\n[`local_cell_level()`](src/timeshap/explainer/cell_level.py) calculates cell level explanations\nof a given sequence with the respective event- and feature-level explanations\nand user-given parameteres, returing the respective cell-level explanations.\n\n[`plot_cell_level()`](src/timeshap/plot/cell_level.py) plots the feature-level explanations\ncalculated by  `local_cell_level()`.\n\n\u003cimg src=\"resources/images/cell_level.png\" width=\"350\"\u003e\n\n##### Local Report\n\n[`local_report()`](src/timeshap/explainer/local_methods.py) calculates TimeSHAP \nlocal explanations for a given sequence and plots them.\n\n\u003cimg src=\"resources/images/local_report.png\" width=\"800\"\u003e\n\n#### Global Explanations\n\n\n##### Global pruning statistics\n\n[`prune_all()`](src/timeshap/explainer/pruning.py) performs the pruning\nalgorithm on multiple given sequences.\n\n[`pruning_statistics()`](src/timeshap/plot/pruning.py) calculates the pruning\nstatistics for several user-given pruning tolerances using the pruning\ndata calculated by `prune_all()`, returning a `pandas.DataFrame` with the statistics.\n\n\n##### Global event level explanations\n\n[`event_explain_all()`](src/timeshap/explainer/event_level.py) calculates TimeSHAP \nevent level explanations for multiple instances given user defined parameters.\n\n[`plot_global_event()`](src/timeshap/plot/event_level.py) plots the global event-level explanations\ncalculated by `event_explain_all()`.\n\n\u003cimg src=\"resources/images/global_event.png\" width=\"600\"\u003e\n\n##### Global feature level explanations\n\n[`feat_explain_all()`](src/timeshap/explainer/feature_level.py) calculates TimeSHAP \nfeature level explanations for multiple instances given user defined parameters.\n\n[`plot_global_feat()`](src/timeshap/plot/feature_level.py) plots the global feature-level \nexplanations calculated by `feat_explain_all()`.\n\n\u003cimg src=\"resources/images/global_feat.png\" width=\"450\"\u003e\n\n\n##### Global report\n[`global_report()`](src/timeshap/explainer/global_methods.py) calculates TimeSHAP \nexplanations for multiple instances, aggregating the explanations on two plots\nand returning them.\n\n\u003cimg src=\"resources/images/global_report.png\" width=\"800\"\u003e\n\n\n\n## Tutorial\nIn order to demonstrate TimeSHAP interfaces and methods, you can consult\n[AReM.ipynb](notebooks/AReM/AReM.ipynb). \nIn this tutorial we get an open-source dataset, process it, train \nPytorch recurrent model with it and use TimeSHAP to explain it, showcasing all \npreviously described methods.\n\nAdditionally, we also train a TensorFlow model on the same dataset \n[AReM_TF.ipynb](notebooks/AReM/AReM_TF.ipynb).\n\n## Repository Structure\n\n- [`notebooks`](notebooks) - tutorial notebooks demonstrating the package;\n- [`src/timeshap`](src/timeshap) - the package source code;\n  - [`src/timeshap/explainer`](src/timeshap/explainer) - TimeSHAP methods to produce the explanations\n  - [`src/timeshap/explainer/kernel`](src/timeshap/explainer/kernel) - TimeSHAPKernel\n  - [`src/timeshap/plot`](src/timeshap/plot) - TimeSHAP methods to produce explanation plots\n  - [`src/timeshap/utils`](src/timeshap/utils) - util methods for TimeSHAP execution\n  - [`src/timeshap/wrappers`](src/timeshap/wrappers) - Wrapper classes for models in order to ease TimeSHAP explanations\n\n## Citing TimeSHAP\n```\n@inproceedings{bento2021timeshap,\n    author = {Bento, Jo\\~{a}o and Saleiro, Pedro and Cruz, Andr\\'{e} F. and Figueiredo, M\\'{a}rio A.T. and Bizarro, Pedro},\n    title = {TimeSHAP: Explaining Recurrent Models through Sequence Perturbations},\n    year = {2021},\n    isbn = {9781450383325},\n    publisher = {Association for Computing Machinery},\n    address = {New York, NY, USA},\n    url = {https://doi.org/10.1145/3447548.3467166},\n    doi = {10.1145/3447548.3467166},\n    booktitle = {Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery \u0026 Data Mining},\n    pages = {2565–2573},\n    numpages = {9},\n    keywords = {SHAP, Shapley values, TimeSHAP, XAI, RNN, explainability},\n    location = {Virtual Event, Singapore},\n    series = {KDD '21}\n}\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffeedzai%2Ftimeshap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffeedzai%2Ftimeshap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffeedzai%2Ftimeshap/lists"}