{"id":28411217,"url":"https://github.com/capitalone/ablation","last_synced_at":"2025-09-01T03:33:51.569Z","repository":{"id":58932646,"uuid":"525805936","full_name":"capitalone/ablation","owner":"capitalone","description":"Evaluating XAI methods through ablation studies.","archived":false,"fork":false,"pushed_at":"2024-12-28T21:57:33.000Z","size":40295,"stargazers_count":17,"open_issues_count":1,"forks_count":7,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-08-19T06:06:23.286Z","etag":null,"topics":["ablation","explainability","ground-truth","xai"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2207.05566","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/capitalone.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":"roadmap.md","authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-17T13:32:50.000Z","updated_at":"2025-04-26T23:15:37.000Z","dependencies_parsed_at":"2024-12-23T23:31:28.999Z","dependency_job_id":null,"html_url":"https://github.com/capitalone/ablation","commit_stats":{"total_commits":15,"total_committers":4,"mean_commits":3.75,"dds":"0.19999999999999996","last_synced_commit":"4530a2dabb51805af4f68e91e9850c1aad52b012"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/capitalone/ablation","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/capitalone%2Fablation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/capitalone%2Fablation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/capitalone%2Fablation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/capitalone%2Fablation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/capitalone","download_url":"https://codeload.github.com/capitalone/ablation/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/capitalone%2Fablation/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273069599,"owners_count":25040078,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-01T02:00:09.058Z","response_time":120,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ablation","explainability","ground-truth","xai"],"created_at":"2025-06-02T14:44:44.967Z","updated_at":"2025-09-01T03:33:51.538Z","avatar_url":"https://github.com/capitalone.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Ablation\n\nA library to assess the effectiveness of XAI methods through ablation.\n\nFor API documentation, visit the [documentation page](https://capitalone.github.io/ablation/).\n\n### Background\n\nExplainable artificial intelligence (XAI) methods lack ground truth.  In its place, method developers have relied on axioms to determine desirable properties for their explanations behavior.  For high stakes uses of machine learning that require explainability, it is not sufficient to rely on axioms, as the implementation, or its usage, can fail to live up to the ideal.  A procedure frequently used to assess their utility, and to some extent their fidelity, is an *ablation study*.  By perturbing the input variables in rank order of importance, the goal is to assess the sensitivity of the model's performance.\n\nThis implementation can be used to reproduce the experiments in [BASED-XAI: Breaking Ablation Studies Down for Explainable Artificial Intelligence](https://arxiv.org/abs/2207.05566).\n\n### Installation\n\nPip\n\n```sh\nconda create -n ablation python=3.11 --yes\nsource activate ablation\npip install ablation\n```\n\n### Contributions\n\nIf you would like to contribute, please download the code from `master` to ensure any contributions are made towards vulnerability free code.\n\nInstall the development dependencies:\n```\npip install -e .\"[dev]\"\n```\n\nInstall `pre-commit` hooks:\n\n```\npre-commit install\npre-commit run\n```\n\n### Get Started\n\nConfigure experiment yaml file:\n\n```yaml\ndataset_name: synthetic_cat             # dataset name\nmodel_type: nn                          # model type (nn/linear)\npath: synthetic_cat                     # experiment path\nload: false                             # load experiment from path\nrerun_ablation: false                # rerun ablation if loaded\n\nn_trials: 4                             # number of trials\ndataset_sample_perc: 1                  # fraction of dataset to subsample\ndataset_n_random_features: 4            # random features to add\n\nexplanation_methods:                    # explanation methods\n  - random\n  - deep_shap\n  - kernel_shap\nperturbation_config:                    # perturbation methods and dict of params\n  constant_median: {}\n\nbaseline_config:                        # baseline methods and dict of params\n  constant_median: {}\n  training:\n    nsamples: 50\n\n\nablation_args:                          # ablation arguments\n  scoring_methods: [log_loss, auroc]    # performance metric\n  local: true                           # local vs global explanations\n\n```\n\nRun experiment\n```Python\nfrom ablation.experiment import Config, Experiment\n\nconfig = Config.from_yaml_file(\"config.yml\")\nexp = Experiment(config)\nresults = exp.run_exp()\n```\n### Replicate experiments\n\nExperiments for the paper used `pytorch_lightning==1.5.7`. Due to vulnerabilities, we needed to update to a more recent version.  In order to accurately reproduce the experiments in the paper, install release `v0.1.0` and revert to the previous version of pytorch lightning:\n```\npip install ablation==0.1.0\npip install pytorch_lightning==1.5.7\n```\n\nTo replicate KDD experiments for all real datasets run the following\n```sh\ncd kdd_configs\npython run_exp.py\n```\n\nTo replicate KDD experiments for synthetic datasets run the following\n```sh\ncd kdd_configs\npython run_exp_synthetic.py\n```\n\n### Tests\n\nTo run tests:\n```sh\npython -m pytest tests/unittests\n```\n\n### Citing\nIf you use `ablation` in your work, please cite our paper:\n\n```\n@inproceedings{hameed:basedxai:2022,\n  author    = {Hameed, Isha and Sharpe, Samuel and Barcklow, Daniel and Au-Yeung, Justin and Verma, Sahil and Huang, Jocelyn and Barr, Brian and Bruss, C. Bayan},\n  title     = {{BASED-XAI: Breaking Ablation Studies Down for Explainable Artificial Intelligence}},\n  year      = {2022},\n  maintitle = {ACM SIGKDD International Conference on Knowledge Discovery and Data Mining},\n  booktitle = {Workshop on Machine Learning in Finance},\n}\n```\n\n### Contributors\n\nWe welcome Your interest in Capital One’s Open Source Projects (the\n“Project”). Any Contributor to the Project must accept and sign an\nAgreement indicating agreement to the license terms below. Except for\nthe license granted in this Agreement to Capital One and to recipients\nof software distributed by Capital One, You reserve all right, title,\nand interest in and to Your Contributions; this Agreement does not\nimpact Your rights to use Your own Contributions for any other purpose.\n\n[Sign the Individual Agreement](https://docs.google.com/forms/d/19LpBBjykHPox18vrZvBbZUcK6gQTj7qv1O5hCduAZFU/viewform)\n\n[Sign the Corporate Agreement](https://docs.google.com/forms/d/e/1FAIpQLSeAbobIPLCVZD_ccgtMWBDAcN68oqbAJBQyDTSAQ1AkYuCp_g/viewform?usp=send_form)\n\n\n### Code of Conduct\n\nThis project adheres to the [Open Code of Conduct](https://developer.capitalone.com/resources/code-of-conduct)\nBy participating, you are\nexpected to honor this code.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcapitalone%2Fablation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcapitalone%2Fablation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcapitalone%2Fablation/lists"}