https://github.com/statmlben/dnn-inference

Significance tests of feature relevance for a black-box learner
https://github.com/statmlben/dnn-inference

black-box-testing feature-importance feature-selection hypothesis-testing image-inference significance-testing statistical-inference statistical-tests xai

Last synced: 5 months ago
JSON representation

Significance tests of feature relevance for a black-box learner

• Host: GitHub
• URL: https://github.com/statmlben/dnn-inference
• Owner: statmlben
• License: mit
• Created: 2020-02-18T01:32:58.000Z (over 6 years ago)
• Default Branch: master
• Last Pushed: 2024-08-02T16:37:37.000Z (almost 2 years ago)
• Last Synced: 2025-12-28T13:56:17.462Z (5 months ago)
• Topics: black-box-testing, feature-importance, feature-selection, hypothesis-testing, image-inference, significance-testing, statistical-inference, statistical-tests, xai
• Language: Python
• Homepage: https://dnn-inference.readthedocs.io
• Size: 568 MB
• Stars: 20
• Watchers: 0
• Forks: 4
• Open Issues: 2
• Metadata Files:
  • Readme: README.rst
  • License: LICENSE

Awesome Lists containing this project

README

          
.. dnn-inference documentation master file

.. raw:: html

    

        

    

🔬 dnn inference

==================

.. -*- mode: rst -*-

|PyPi|_ |Keras|_ |MIT|_ |Python3|_ |tensorflow|_ |downloads|_

.. |PyPi| image:: https://badge.fury.io/py/dnn-inference.svg

.. _PyPi: https://pypi.org/project/dnn-inference/

.. |Keras| image:: https://img.shields.io/badge/keras-tf.keras-red.svg

.. _Keras: https://keras.io/

.. |MIT| image:: https://img.shields.io/pypi/l/dnn-inference.svg

.. _MIT: https://opensource.org/licenses/MIT

.. |Python3| image:: https://img.shields.io/badge/python-3-green.svg

.. _Python3: www.python.org

.. |tensorflow| image:: https://img.shields.io/badge/keras-tensorflow-blue.svg

.. _tensorflow: https://www.tensorflow.org/

.. |downloads| image:: https://pepy.tech/badge/dnn-inference

.. _downloads: https://pepy.tech/project/dnn-inference

.. .. image:: ./logo/logo_header.png

..    :width: 900

**dnn_inference** is a Python module for hypothesis testing based on black-box models, including **deep neural networks**. 

- GitHub repo: `https://github.com/statmlben/dnn-inference `_

- Documentation: `https://dnn-inference.readthedocs.io `_

- PyPi: `https://pypi.org/project/dnn-inference `_

- Open Source: `MIT license `_

- Paper: `arXiv:2103.04985 `_

🎯 What We Can Do

-----------------

**dnn-inference** is able to provide an asymptotically valid `p-value` to examine if $S$ is discriminative features to predict $Y$.

Specifically, the proposed testing is:

$$H_0: R(f^*) - R_{S}(g^*) = 0,   H_a: R(f^*) - R_{S}(g^*) < 0,$$

where $X_S$ is a collection of hypothesized features, $R$ and $R_S$ are risk functions with/without the hypothesized features $X_S$, 

and $f^*$ and $g^*$ are population minimizers on $R$ and $R_S$ respectively. 

The proposed test just considers the difference between the best predictive scores with/without hypothesized features. 

Please check more details in our paper `arXiv:2103.04985 `_.

- When `log-likelihood` is used as a loss function, then the test is equivalent to a conditional independence test: `$Y indep X_{S} | X_{S^c}$`. 

- Only `a small number of fitting` on neural networks is required, and the number can be as small as 1.

- Asymptotically Type I error control and power consistency.

Installation

============

Dependencies

------------

``dnn-inference`` requires: **Python>=3.8** + `requirements.txt <./requirements.txt>`_

.. code:: bash

  pip install -r requirements.txt

User installation

-----------------

Install ``dnn-inference`` using ``pip``

.. code:: bash

	pip install dnn_inference

	pip install git+https://github.com/statmlben/dnn-inference.git

Reference

---------

**If you use this code please star the repository and cite the following paper:**

.. code:: bib

   @article{dai2022significance,

      title={Significance Tests of Feature Relevance for a Black-Box Learner},

      author={Dai, Ben and Shen, Xiaotong and Pan, Wei},

      journal={IEEE Transactions on Neural Networks and Learning Systems},

      year={2022},

      publisher={IEEE}

   }

Notebook

========

- **MNIST dataset**: `Notebook1 `_

- **Boston house prices dataset**: `Notebook2 `_