{"id":29182338,"url":"https://github.com/openvinotoolkit/openvino_xai","last_synced_at":"2026-03-15T09:40:08.522Z","repository":{"id":244036067,"uuid":"806947463","full_name":"openvinotoolkit/openvino_xai","owner":"openvinotoolkit","description":"OpenVINO™ Explainable AI (XAI) Toolkit: Visual Explanation for OpenVINO Models","archived":false,"fork":false,"pushed_at":"2024-10-04T01:25:14.000Z","size":10898,"stargazers_count":29,"open_issues_count":3,"forks_count":9,"subscribers_count":5,"default_branch":"develop","last_synced_at":"2025-01-03T09:15:13.151Z","etag":null,"topics":["explainable-ai","openvino-toolkit","xai"],"latest_commit_sha":null,"homepage":"https://openvinotoolkit.github.io/openvino_xai/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openvinotoolkit.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-28T07:53:25.000Z","updated_at":"2024-11-21T20:40:36.000Z","dependencies_parsed_at":"2024-06-28T08:46:45.162Z","dependency_job_id":"528a4b5d-0226-4c4d-977f-0f47cb5fd313","html_url":"https://github.com/openvinotoolkit/openvino_xai","commit_stats":null,"previous_names":["openvinotoolkit/openvino_xai"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/openvinotoolkit/openvino_xai","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openvinotoolkit%2Fopenvino_xai","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openvinotoolkit%2Fopenvino_xai/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openvinotoolkit%2Fopenvino_xai/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openvinotoolkit%2Fopenvino_xai/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openvinotoolkit","download_url":"https://codeload.github.com/openvinotoolkit/openvino_xai/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openvinotoolkit%2Fopenvino_xai/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263029206,"owners_count":23402354,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["explainable-ai","openvino-toolkit","xai"],"created_at":"2025-07-01T20:03:40.826Z","updated_at":"2026-03-15T09:40:03.481Z","avatar_url":"https://github.com/openvinotoolkit.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# OpenVINO™ Explainable AI Toolkit - OpenVINO XAI\n\n---\n\n[Features](#features) •\n[Install](#installation) •\n[Quick Start](#quick-start) •\n[License](#license) •\n[Documentation](https://openvinotoolkit.github.io/openvino_xai/releases/1.1.0)\n\n![Python](https://img.shields.io/badge/python-3.10%2B-green)\n[![OpenVINO](https://img.shields.io/badge/openvino-2024.4-purple)](https://pypi.org/project/openvino/)\n[![codecov](https://codecov.io/gh/openvinotoolkit/openvino_xai/graph/badge.svg?token=NR0Z0CWDK9)](https://codecov.io/gh/openvinotoolkit/openvino_xai)\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n[![PyPI](https://img.shields.io/pypi/v/openvino_xai)](https://pypi.org/project/openvino_xai)\n[![Downloads](https://static.pepy.tech/badge/openvino_xai)](https://pepy.tech/project/openvino_xai)\n\n---\n\n\u003c/div\u003e\n\n![OpenVINO XAI Concept](docs/source/_static/ovxai-concept.svg)\n\n**OpenVINO™ Explainable AI (XAI) Toolkit** provides a suite of XAI algorithms for visual explanation of\n[**OpenVINO™**](https://github.com/openvinotoolkit/openvino) as well as [**PyTorch**](https://pytorch.org) and [**ONNX**](https://onnx.ai/) models.\n\nGiven AI models and input images, **OpenVINO XAI** generates **saliency maps**\nwhich highlights regions of the interest in the inputs from the models' perspective\nto help users understand the reason why the complex AI models output such responses.\n\nUsing this package, you can augment the model analysis \u0026 explanation feature\non top of the existing OpenVINO inference pipeline with a few lines of code.\n\n```python\nimport openvino_xai as xai\n\nexplainer = xai.Explainer(model=ov_model, task=xai.Task.CLASSIFICATION)\n\n# Existing inference pipeline\nfor i, image in enumerate(images):\n    labels = infer(model=ov_model, image=image)\n\n    # Model analysis\n    explanation = explainer(data=image, targets=labels)\n    explanation.save(dir_path=\"./xai\", name=str(i))\n```\n\n---\n\n## Features\n\n### What's new in v1.1.0\n\n* Support PyTorch models with `insert_xai()` API for saliency map generation on PyTorch / ONNX runtime\n* Support OpenVINO IR (.xml) / ONNX (.onnx) model files for `Explainer`\n* Enable AISE method: Adaptive Input Sampling for Explanation of Black-box Models\n* Add Pointing Game, Insertion-Deletion AUC and ADCC quality metrics for saliency maps\n* Upgrade OpenVINO to 2024.4.0\n* Add saliency map visualization with explanation.plot()\n* Enable flexible naming for saved saliency maps and include confidence scores\n\nPlease refer to the [change logs](CHANGELOG.md) for the full release history.\n\n### Supported XAI methods\n\nAt the moment, *Image Classification* and *Object Detection* tasks are supported for the *Computer Vision* domain.\n*Black-Box* (model agnostic but slow) methods and *White-Box* (model specific but fast) methods are supported:\n\n| Domain          | Task                 | Type      | Algorithm              | Links |\n|-----------------|----------------------|-----------|------------------------|-------|\n| Computer Vision | Image Classification | White-Box | ReciproCAM             | [paper](https://openaccess.thecvf.com/content/CVPR2024W/XAI4CV/papers/Byun_ReciproCAM_Lightweight_Gradient-free_Class_Activation_Map_for_Post-hoc_Explanations_CVPRW_2024_paper.pdf) / [src](openvino_xai/me4thods/white_box/recipro_cam.py) |\n|                 |                      |           | VITReciproCAM          | [paper](https://arxiv.org/abs/2310.02588) / [src](openvino_xai/methods/white_box/recipro_cam.py) |\n|                 |                      |           | ActivationMap          | experimental / [src](openvino_xai/methods/white_box/activation_map.py)                           |\n|                 |                      | Black-Box | AISEClassification     | [src](openvino_xai/methods/black_box/aise/classification.py)                                     |\n|                 |                      |           | RISE                   | [paper](https://arxiv.org/abs/1806.07421v3) / [src](openvino_xai/methods/black_box/rise.py)      |\n|                 | Object Detection     | White-Box | DetClassProbabilityMap | experimental / [src](openvino_xai/methods/white_box/det_class_probability_map.py)                |\n|                 |                      | Black-Box | AISEDetection          | [src](openvino_xai/methods/black_box/aise/detection.py)                                          |\n\nSee more method comparison at the [User Guide](docs/source/user-guide.md).\n\n### Supported explainable models\n\nMost of CNNs and Transformer models from [Pytorch Image Models (timm)](https://github.com/huggingface/pytorch-image-models) are supported and validated.\n\nPlease refer to the following known issues for unsupported models and reasons.\n\n* [Runtime error from ONNX / OpenVINO IR models while conversion or inference for XAI (#29)](https://github.com/openvinotoolkit/openvino_xai/issues/29)\n* [Models not supported by white box XAI methods (#30)](https://github.com/openvinotoolkit/openvino_xai/issues/30)\n\n\u003e **_NOTE:_**  GenAI / LLMs would be also supported incrementally in the upcoming releases.\n\n---\n\n## Installation\n\n\u003e **_NOTE:_**  OpenVINO XAI works on Python 3.10 or higher\n\n\u003cdetails\u003e\n\u003csummary\u003eSet up environment\u003c/summary\u003e\n\n```bash\n# Create virtual env.\npython3.10 -m venv .ovxai\n\n# Activate virtual env.\nsource .ovxai/bin/activate\n```\n\u003c/details\u003e\n\nInstall from PyPI package\n\n```bash\n# Base package (for normal use):\npip install openvino_xai\n\n# Dev package (for development):\npip install openvino_xai[dev]\n```\n\n\u003cdetails\u003e\n\u003csummary\u003eInstall from source\u003c/summary\u003e\n\n```bash\n# Clone the source repository\ngit clone https://github.com/openvinotoolkit/openvino_xai.git\ncd openvino_xai\n\n# Editable mode (for development):\npip install -e .[dev]\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e(Optional) Enable PyTorch support\u003c/summary\u003e\n\nYou can enjoy the PyTorch XAI feature if the PyTorch is installed along with the OpenVINO XAI.\n\n```bash\n# Install PyTorch (CPU version as example)\npip3 install torch --index-url https://download.pytorch.org/whl/cpu\n```\nPlease refer to the [PyTorch Installation Guide](https://pytorch.org/get-started/locally/) for other options.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eVerify installation\u003c/summary\u003e\n\n```bash\n# Run tests\npytest -v -s ./tests/unit\n\n# Run code quality checks\npre-commit run --all-files\n```\n\u003c/details\u003e\n\n---\n\n## Quick Start\n\n### Hello, OpenVINO XAI\n\nLet's imagine the case that our OpenVINO model is up and running on a inference pipeline.\nWhile watching the outputs, we may want to analyze the model's behavior for debugging or understanding purposes.\n\nBy using the **OpenVINO XAI** `Explainer`, we can visualize why the model gives such responses.\nIn this example, we are trying to know the reason why the model outputs a `cheetah` label for the given input image.\n\n```python\nimport cv2\nimport numpy as np\nimport openvino as ov\nimport openvino_xai as xai\n\n# Load the model: IR or ONNX\nov_model: ov.Model = ov.Core().read_model(\"mobilenet_v3.xml\")\n\n# Load the image to be analyzed\nimage: np.ndarray = cv2.imread(\"tests/assets/cheetah_person.jpg\")\nimage = cv2.resize(image, dsize=(224, 224))\nimage = np.expand_dims(image, 0)\n\n# Create the Explainer for the model\nexplainer = xai.Explainer(\n    model=ov_model,  # accepts path arguments \"mobilenet_v3.xml\" or \"mobilenet_v3.onnx\" as well\n    task=xai.Task.CLASSIFICATION,\n)\n\n# Generate saliency map for the label of interest\nexplanation: xai.Explanation = explainer(\n    data=image,\n    targets=293,  # (cheetah), accepts label indices or actual label names if label_names provided\n    overlay=True,  # saliency map overlay over the input image, defaults to False\n)\n\n# Save saliency maps to output directory\nexplanation.save(dir_path=\"./output\")\n```\n\nOriginal image | Explained image\n---------------|----------------\n![Oringinal images](tests/assets/cheetah_person.jpg) | ![Explained image](docs/source/_static/xai-cheetah.png)\n\nWe can see that model is focusing on the body or skin area of the animals to tell if this image contains actual cheetahs.\n\n### Insert XAI head to your models\n\nUsing the `insert_xai` API, we can insert XAI head to existing OpenVINO or PyTorch models directly and get additional \"saliency_map\" output in the same inference pipeline.\n\n```python\nimport torch\nimport timm\n\n# Get a PyTorch model from TIMM\ntorch_model: torch.nn.Module = timm.create_model(\"resnet18.a1_in1k\", in_chans=3, pretrained=True)\n\n# Insert XAI head\nmodel_xai: torch.nn.Module = xai.insert_xai(torch_model, xai.Task.CLASSIFICATION)\n\n# Torch XAI model inference\nmodel_xai.eval()\nwith torch.no_grad():\n    outputs = model_xai(torch.from_numpy(image_norm))\n    logits = outputs[\"prediction\"]  # BxC\n    saliency_maps = outputs[\"saliency_map\"]  # BxCxHxW: per-class saliency map\n```\n\n### More advanced use-cases\n\nUsers could tweak the basic use-case according to their purpose, which include but not limited to:\n\n* Select XAI mode (White-Box or Black-Box) or even specific method which are automatically decided by default\n* Provide custom model pre/post processing functions like resize and normalizations which the model expects\n* Customize output image visualization options\n* Explain multiple class targets, passing them as label indices or as actual label names\n* Call explainer multiple times to explain multiple images or to use different targets\n* Insert XAI head to your PyTorch models and export to ONNX format to generate saliency maps on ONNX Runtime\n  (Refer to the [full example script](./examples/run_torch_onnx.py))\n\nPlease find more options and scenarios in the following links:\n\n* [OpenVINO XAI User Guide](docs/source/user-guide.md)\n* [OpenVINO Notebook - XAI Basic](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/explainable-ai-1-basic/explainable-ai-1-basic.ipynb)\n* [OpenVINO Notebook - XAI Deep Dive](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/explainable-ai-2-deep-dive/explainable-ai-2-deep-dive.ipynb)\n* [OpenVINO Notebook - Saliency Map Interpretation](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/explainable-ai-3-map-interpretation/explainable-ai-3-map-interpretation.ipynb)\n\n### Playing with the examples\n\nPlease look around the runnable [example scripts](./examples) and play with them to get used to the `Explainer` and `insert_xai` APIs.\n\n```bash\n# Prepare models by running tests (need \"pip install openvino_xai[dev]\" extra option)\n# Models are downloaded and stored in .data/otx_models\npytest tests/test_classification.py\n\n# Run a bunch of classification examples\n# All outputs will be stored in the corresponding output directory\npython examples/run_classification.py .data/otx_models/mlc_mobilenetv3_large_voc.xml \\\ntests/assets/cheetah_person.jpg --output output\n\n# Run PyTorch and ONNX support example\npython examples/run_torch_onnx.py\n```\n\n---\n\n## Contributing\n\nFor those who would like to contribute to the library, please refer to the [contribution guide](CONTRIBUTING.md) for details.\n\nPlease let us know via the [Issues tab](https://github.com/openvinotoolkit/openvino_xai/issues/new) if you have any issues, feature requests, or questions.\n\nThank you! We appreciate your support!\n\n\u003ca href=\"https://github.com/openvinotoolkit/openvino_xai/graphs/contributors\"\u003e\n  \u003cimg src=\"https://contrib.rocks/image?repo=openvinotoolkit/openvino_xai\" /\u003e\n\u003c/a\u003e\n\n---\n\n## License\n\nOpenVINO™ Toolkit is licensed under [Apache License Version 2.0](LICENSE).\nBy contributing to the project, you agree to the license and copyright terms therein and release your contribution under these terms.\n\n---\n\n## Disclaimer\n\nIntel is committed to respecting human rights and avoiding complicity in human rights abuses.\nSee Intel's [Global Human Rights Principles](https://www.intel.com/content/www/us/en/policy/policy-human-rights.html).\nIntel's products and software are intended only to be used in applications that do not cause or contribute to a violation of an internationally recognized human right.\n\n---\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenvinotoolkit%2Fopenvino_xai","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenvinotoolkit%2Fopenvino_xai","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenvinotoolkit%2Fopenvino_xai/lists"}