{"id":20453177,"url":"https://github.com/pietrobarbiero/pytorch_explain","last_synced_at":"2025-04-06T22:10:19.230Z","repository":{"id":37443519,"uuid":"356630474","full_name":"pietrobarbiero/pytorch_explain","owner":"pietrobarbiero","description":"PyTorch Explain: Interpretable Deep Learning in Python.","archived":false,"fork":false,"pushed_at":"2024-05-16T12:44:52.000Z","size":44134,"stargazers_count":153,"open_issues_count":1,"forks_count":14,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-03-30T20:11:13.326Z","etag":null,"topics":["deep-learning","entropy","explainability","explainable-ai","interpretability","interpretable-ai","interpretable-deep-learning","interpretable-machine-learning","lens","logic","machine-learning","neural-network","python","pytorch","sympy"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pietrobarbiero.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-10T16:02:36.000Z","updated_at":"2025-03-20T11:29:12.000Z","dependencies_parsed_at":"2024-11-15T11:12:09.358Z","dependency_job_id":null,"html_url":"https://github.com/pietrobarbiero/pytorch_explain","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":"pietrobarbiero/project-template","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pietrobarbiero%2Fpytorch_explain","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pietrobarbiero%2Fpytorch_explain/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pietrobarbiero%2Fpytorch_explain/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pietrobarbiero%2Fpytorch_explain/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pietrobarbiero","download_url":"https://codeload.github.com/pietrobarbiero/pytorch_explain/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247557767,"owners_count":20958047,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","entropy","explainability","explainable-ai","interpretability","interpretable-ai","interpretable-deep-learning","interpretable-machine-learning","lens","logic","machine-learning","neural-network","python","pytorch","sympy"],"created_at":"2024-11-15T11:11:58.844Z","updated_at":"2025-04-06T22:10:19.195Z","avatar_url":"https://github.com/pietrobarbiero.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":".. image:: https://raw.githubusercontent.com/pietrobarbiero/pytorch_explain/master/doc/_static/img/pye_logo_text_dark.svg\n    :align: center\n    :height: 100px\n    :scale: 50 %\n\n\n\n-------------\n\n\n\n|Build|\n|Coverage|\n\n|Docs|\n|Dependendencies|\n\n|PyPI license|\n|PyPI-version|\n\n\n.. |Build| image:: https://img.shields.io/travis/com/pietrobarbiero/pytorch_explain?label=Master%20Build\u0026style=for-the-badge\n    :alt: Travis (.com)\n    :target: https://app.travis-ci.com/github/pietrobarbiero/pytorch_explain\n\n.. |Coverage| image:: https://img.shields.io/codecov/c/gh/pietrobarbiero/pytorch_explain?label=Test%20Coverage\u0026style=for-the-badge\n    :alt: Codecov\n    :target: https://codecov.io/gh/pietrobarbiero/pytorch_explain\n\n.. |Docs| image:: https://img.shields.io/readthedocs/pytorch_explain/latest?style=for-the-badge\n    :alt: Read the Docs (version)\n    :target: https://pytorch_explain.readthedocs.io/en/latest/\n\n.. |Dependendencies| image:: https://img.shields.io/requires/github/pietrobarbiero/pytorch_explain?style=for-the-badge\n    :alt: Requires.io\n    :target: https://requires.io/github/pietrobarbiero/pytorch_explain/requirements/?branch=master\n\n.. |PyPI license| image:: https://img.shields.io/pypi/l/torch_explain.svg?style=for-the-badge\n   :target: https://pypi.org/project/torch-explain/\n\n.. |PyPI-version| image:: https://img.shields.io/pypi/v/torch_explain?style=for-the-badge\n    :alt: PyPI\n    :target: https://pypi.org/project/torch-explain/\n\n.. image:: https://zenodo.org/badge/356630474.svg\n   :target: https://zenodo.org/badge/latestdoi/356630474\n\n\n`PyTorch, Explain!` is an extension library for PyTorch to develop\nexplainable deep learning models going beyond the current accuracy-interpretability trade-off.\n\nThe library includes a set of tools to develop:\n\n\n* Deep Concept Reasoner (Deep CoRe): an interpretable concept-based model going\n  **beyond the current accuracy-interpretability trade-off**;\n* Concept Embedding Models (CEMs): a class of concept-based models going\n  **beyond the current accuracy-explainability trade-off**;\n* Logic Explained Networks (LENs): a class of concept-based models generating\n  accurate compound logic explanations for their predictions\n  **without the need for a post-hoc explainer**.\n\nTable of Content\n-----------------\n* `Quick start \u003chttps://github.com/pietrobarbiero/pytorch_explain#quick-start\u003e`_\n* `Quick tutorial on Concept Embedding Models \u003chttps://github.com/pietrobarbiero/pytorch_explain#quick-tutorial-on-concept-embedding-models\u003e`_\n* `Quick tutorial on Deep Concept Reasoning \u003chttps://github.com/pietrobarbiero/pytorch_explain#quick-tutorial-on-deep-concept-reasoning\u003e`_\n* `Quick tutorial on Logic Explained Networks \u003chttps://github.com/pietrobarbiero/pytorch_explain#quick-tutorial-on-logic-explained-networks\u003e`_\n* `Benchmark datasets \u003chttps://github.com/pietrobarbiero/pytorch_explain#benchmark-datasets\u003e`_\n* `Theory \u003chttps://github.com/pietrobarbiero/pytorch_explain#theory\u003e`_\n* `Authors \u003chttps://github.com/pietrobarbiero/pytorch_explain#authors\u003e`_\n* `Licence \u003chttps://github.com/pietrobarbiero/pytorch_explain#licence\u003e`_\n\nQuick start\n---------------\n\nYou can install ``torch_explain`` along with all its dependencies from\n`PyPI \u003chttps://pypi.org/project/torch_explain/\u003e`__:\n\n.. code:: bash\n\n    pip install torch-explain\n\n\nQuick tutorial on Concept Embedding Models\n-----------------------------------------------\n\nUsing concept embeddings we can solve concept-based problems very efficiently!\nFor this simple tutorial, let's approach the XOR benchmark dataset:\n\n.. code:: python\n\n    import torch\n    import torch_explain as te\n    from torch_explain import datasets\n    from sklearn.metrics import accuracy_score\n    from sklearn.model_selection import train_test_split\n\n    x, c, y = datasets.xor(500)\n    x_train, x_test, c_train, c_test, y_train, y_test = train_test_split(x, c, y, test_size=0.33, random_state=42)\n\nWe just need to define a task predictor and a concept encoder using a\nconcept embedding layer:\n\n.. code:: python\n\n    import torch\n    import torch_explain as te\n\n    embedding_size = 8\n    concept_encoder = torch.nn.Sequential(\n        torch.nn.Linear(x.shape[1], 10),\n        torch.nn.LeakyReLU(),\n        te.nn.ConceptEmbedding(10, c.shape[1], embedding_size),\n    )\n    task_predictor = torch.nn.Sequential(\n        torch.nn.Linear(c.shape[1]*embedding_size, 1),\n    )\n    model = torch.nn.Sequential(concept_encoder, task_predictor)\n\nWe can now train the network by optimizing the cross entropy loss\non concepts and tasks:\n\n.. code:: python\n\n    optimizer = torch.optim.AdamW(model.parameters(), lr=0.01)\n    loss_form_c = torch.nn.BCELoss()\n    loss_form_y = torch.nn.BCEWithLogitsLoss()\n    model.train()\n    for epoch in range(501):\n        optimizer.zero_grad()\n\n        # generate concept and task predictions\n        c_emb, c_pred = concept_encoder(x_train)\n        y_pred = task_predictor(c_emb.reshape(len(c_emb), -1))\n\n        # compute loss\n        concept_loss = loss_form_c(c_pred, c_train)\n        task_loss = loss_form_y(y_pred, y_train)\n        loss = concept_loss + 0.5*task_loss\n\n        loss.backward()\n        optimizer.step()\n\nOnce trained we can check the performance of the model on the test set:\n\n.. code:: python\n\n    c_emb, c_pred = concept_encoder.forward(x_test)\n    y_pred = task_predictor(c_emb.reshape(len(c_emb), -1))\n\n    task_accuracy = accuracy_score(y_test, y_pred \u003e 0)\n    concept_accuracy = accuracy_score(c_test, c_pred \u003e 0.5)\n\nAs you can see the performance of the model is now great as the task\ntask accuracy is around ~100%.\n\n\nQuick tutorial on Deep Concept Reasoning\n-----------------------------------------------\n\nUsing deep concept reasoning we can solve the same problem as above,\nbut with an intrinsically interpretable model! In fact, Deep Concept Reasoners (Deep CoRes)\nmake task predictions by means of interpretable logic rules using concept embeddings.\n\nUsing the same example as before, we can just change the task predictor\nusing a Deep CoRe layer:\n\n.. code:: python\n\n    from torch_explain.nn.concepts import ConceptReasoningLayer\n    import torch.nn.functional as F\n\n    y_train = F.one_hot(y_train.long().ravel()).float()\n    y_test = F.one_hot(y_test.long().ravel()).float()\n\n    task_predictor = ConceptReasoningLayer(embedding_size, y_train.shape[1])\n    model = torch.nn.Sequential(concept_encoder, task_predictor)\n\n\nWe can now train the network by optimizing the cross entropy loss\non concepts and tasks:\n\n.. code:: python\n\n    optimizer = torch.optim.AdamW(model.parameters(), lr=0.01)\n    loss_form = torch.nn.BCELoss()\n    model.train()\n    for epoch in range(501):\n        optimizer.zero_grad()\n\n        # generate concept and task predictions\n        c_emb, c_pred = concept_encoder(x_train)\n        y_pred = task_predictor(c_emb, c_pred)\n\n        # compute loss\n        concept_loss = loss_form(c_pred, c_train)\n        task_loss = loss_form(y_pred, y_train)\n        loss = concept_loss + 0.5*task_loss\n\n        loss.backward()\n        optimizer.step()\n\nOnce trained the Deep CoRe layer can explain its predictions by\nproviding both local and global logic rules:\n\n\n.. code:: python\n\n    local_explanations = task_predictor.explain(c_emb, c_pred, 'local')\n    global_explanations = task_predictor.explain(c_emb, c_pred, 'global')\n\n\nFor global explanations, the reasoner will return a dictionary with entries such as\n``{'class': 'y_0', 'explanation': '~c_0 \u0026 ~c_1', 'count': 94}``, specifying\nfor each logic rule, the task it is associated with and the number of samples\nassociated with the explanation.\n\n\nQuick tutorial on Logic Explained Networks\n---------------------------------------------\n\nFor this simple experiment, let's solve the XOR problem\n(augmented with 100 dummy features):\n\n.. code:: python\n\n    import torch\n    import torch_explain as te\n    from torch.nn.functional import one_hot\n\n    x0 = torch.zeros((4, 100))\n    x_train = torch.tensor([\n        [0, 0],\n        [0, 1],\n        [1, 0],\n        [1, 1],\n    ], dtype=torch.float)\n    x_train = torch.cat([x_train, x0], dim=1)\n    y_train = torch.tensor([0, 1, 1, 0], dtype=torch.long)\n    y_train_1h = one_hot(y_train).to(torch.float)\n\nWe can instantiate a simple feed-forward neural network\nwith 3 layers using the ``EntropyLayer`` as the first one:\n\n.. code:: python\n\n    layers = [\n        te.nn.EntropyLinear(x_train.shape[1], 10, n_classes=y_train_1h.shape[1]),\n        torch.nn.LeakyReLU(),\n        torch.nn.Linear(10, 4),\n        torch.nn.LeakyReLU(),\n        torch.nn.Linear(4, 1),\n    ]\n    model = torch.nn.Sequential(*layers)\n\nWe can now train the network by optimizing the cross entropy loss and the\n``entropy_logic_loss`` loss function incorporating the human prior towards\nsimple explanations:\n\n.. code:: python\n\n    optimizer = torch.optim.AdamW(model.parameters(), lr=0.001)\n    loss_form = torch.nn.BCEWithLogitsLoss()\n    model.train()\n    for epoch in range(2001):\n        optimizer.zero_grad()\n        y_pred = model(x_train).squeeze(-1)\n        loss = loss_form(y_pred, y_train_1h) + 0.0001 * te.nn.functional.entropy_logic_loss(model)\n        loss.backward()\n        optimizer.step()\n\nOnce trained we can extract first-order logic formulas describing\nhow the network composed the input features to obtain the predictions:\n\n.. code:: python\n\n    from torch_explain.logic.nn import entropy\n    from torch.nn.functional import one_hot\n\n    y1h = one_hot(y_train)\n    global_explanations, local_explanations = entropy.explain_classes(model, x_train, y_train, c_threshold=0.5, y_threshold=0.)\n\nExplanations will be logic formulas in disjunctive normal form.\nIn this case, the explanation will be ``y=1`` if and only if ``(f1 AND ~f2) OR (f2  AND ~f1)``\ncorresponding to ``f1 XOR f2``.\n\nThe function automatically assesses the quality of logic explanations in terms\nof classification accuracy and rule complexity.\nIn this case the accuracy is 100% and the complexity is 4.\n\n\nBenchmark datasets\n-------------------------\n\nWe provide a suite of 3 benchmark datasets to evaluate the performance of our models\nin the folder `torch_explain/datasets`. These 3 datasets were proposed as benchmarks\nfor concept-based models in the paper \"Concept Embedding Models: Beyond the Accuracy-Explainability Trade-Off\".\n\nReal-world datasets can be downloaded from the links provided in the supplementary material of the paper.\n\n\nTheory\n--------\nTheoretical foundations can be found in the following papers.\n\nDeep Concept Reasoning (recently accepted at ICML-23)::\n\n    @article{barbiero2023interpretable,\n      title={Interpretable Neural-Symbolic Concept Reasoning},\n      author={Barbiero, Pietro and Ciravegna, Gabriele and Giannini, Francesco and Zarlenga, Mateo Espinosa and Magister, Lucie Charlotte and Tonda, Alberto and Lio, Pietro and Precioso, Frederic and Jamnik, Mateja and Marra, Giuseppe},\n      journal={arXiv preprint arXiv:2304.14068},\n      year={2023}\n    }\n\nConcept Embedding Models::\n\n    @article{espinosa2022concept,\n      title={Concept Embedding Models: Beyond the Accuracy-Explainability Trade-Off},\n      author={Espinosa Zarlenga, Mateo and Barbiero, Pietro and Ciravegna, Gabriele and Marra, Giuseppe and Giannini, Francesco and Diligenti, Michelangelo and Shams, Zohreh and Precioso, Frederic and Melacci, Stefano and Weller, Adrian and others},\n      journal={Advances in Neural Information Processing Systems},\n      volume={35},\n      pages={21400--21413},\n      year={2022}\n    }\n\n\nLogic Explained Networks::\n\n    @article{ciravegna2023logic,\n      title={Logic explained networks},\n      author={Ciravegna, Gabriele and Barbiero, Pietro and Giannini, Francesco and Gori, Marco and Li{\\'o}, Pietro and Maggini, Marco and Melacci, Stefano},\n      journal={Artificial Intelligence},\n      volume={314},\n      pages={103822},\n      year={2023},\n      publisher={Elsevier}\n    }\n\nEntropy-based LENs::\n\n    @inproceedings{barbiero2022entropy,\n      title={Entropy-based logic explanations of neural networks},\n      author={Barbiero, Pietro and Ciravegna, Gabriele and Giannini, Francesco and Li{\\'o}, Pietro and Gori, Marco and Melacci, Stefano},\n      booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},\n      volume={36},\n      number={6},\n      pages={6046--6054},\n      year={2022}\n    }\n\nPsi network (\"learning of constraints\")::\n\n    @inproceedings{ciravegna2020constraint,\n      title={A constraint-based approach to learning and explanation},\n      author={Ciravegna, Gabriele and Giannini, Francesco and Melacci, Stefano and Maggini, Marco and Gori, Marco},\n      booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},\n      volume={34},\n      number={04},\n      pages={3658--3665},\n      year={2020}\n    }\n\n\nLearning with constraints::\n\n    @inproceedings{marra2019lyrics,\n      title={LYRICS: A General Interface Layer to Integrate Logic Inference and Deep Learning},\n      author={Marra, Giuseppe and Giannini, Francesco and Diligenti, Michelangelo and Gori, Marco},\n      booktitle={Joint European Conference on Machine Learning and Knowledge Discovery in Databases},\n      pages={283--298},\n      year={2019},\n      organization={Springer}\n    }\n\nConstraints theory in machine learning::\n\n    @book{gori2017machine,\n      title={Machine Learning: A constraint-based approach},\n      author={Gori, Marco},\n      year={2017},\n      publisher={Morgan Kaufmann}\n    }\n\n\nAuthors\n-------\n\n* `Pietro Barbiero \u003chttp://www.pietrobarbiero.eu/\u003e`__, University of Cambridge, UK.\n* Mateo Espinosa Zarlenga, University of Cambridge, UK.\n* Giuseppe Marra, Katholieke Universiteit Leuven, BE.\n* Steve Azzolin, University of Trento, IT.\n* Francesco Giannini, University of Florence, IT.\n* Gabriele Ciravegna, University of Florence, IT.\n* Dobrik Georgiev, University of Cambridge, UK.\n\n\nLicence\n-------\n\nCopyright 2020 Pietro Barbiero, Mateo Espinosa Zarlenga, Giuseppe Marra,\nSteve Azzolin, Francesco Giannini, Gabriele Ciravegna, and Dobrik Georgiev.\n\nLicensed under the Apache License, Version 2.0 (the \"License\"); you may\nnot use this file except in compliance with the License. You may obtain\na copy of the License at: http://www.apache.org/licenses/LICENSE-2.0.\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n\nSee the License for the specific language governing permissions and\nlimitations under the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpietrobarbiero%2Fpytorch_explain","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpietrobarbiero%2Fpytorch_explain","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpietrobarbiero%2Fpytorch_explain/lists"}