{"id":19505281,"url":"https://github.com/interpretml/talktoebm","last_synced_at":"2025-05-07T19:49:10.316Z","repository":{"id":184281587,"uuid":"669314278","full_name":"interpretml/TalkToEBM","owner":"interpretml","description":"A Natural Language Interface to Explainable Boosting Machines","archived":false,"fork":false,"pushed_at":"2024-07-05T11:40:09.000Z","size":30585,"stargazers_count":66,"open_issues_count":0,"forks_count":12,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-24T05:46:24.611Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://interpret.ml/TalkToEBM/api_reference.html","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/interpretml.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-21T22:36:35.000Z","updated_at":"2025-04-12T17:48:06.000Z","dependencies_parsed_at":"2024-05-28T14:54:13.085Z","dependency_job_id":null,"html_url":"https://github.com/interpretml/TalkToEBM","commit_stats":null,"previous_names":["interpretml/talktoebm"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/interpretml%2FTalkToEBM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/interpretml%2FTalkToEBM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/interpretml%2FTalkToEBM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/interpretml%2FTalkToEBM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/interpretml","download_url":"https://codeload.github.com/interpretml/TalkToEBM/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252948874,"owners_count":21830142,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T22:29:29.562Z","updated_at":"2025-05-07T19:49:10.274Z","avatar_url":"https://github.com/interpretml.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TalkToEBM\n[![Documentation](https://img.shields.io/badge/Documentation-View-blue)](http://interpret.ml/TalkToEBM/api_reference.html)\n![python](https://img.shields.io/badge/python-3.8+-blue)\n![Package Version](https://img.shields.io/pypi/v/t2ebm.svg?style=flat-square)\n![License](https://img.shields.io/github/license/interpretml/TalkToEBM.svg?style=flat-square)\n[![tests](https://github.com/interpretml/TalkToEBM/actions/workflows/run-pytest.yml/badge.svg)](https://github.com/interpretml/TalkToEBM/actions/workflows/run-pytest.yml)\n[![Downloads](https://pepy.tech/badge/t2ebm)](https://pepy.tech/project/t2ebm)\n\u003cbr/\u003e\n\n\u003e ### A Natural Language Interface to Explainable Boosting Machines\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"images/landing.png\" alt=\"drawing\" width=\"900\"/\u003e\n\u003c/p\u003e\n\nTalkToEBM is an open-source package that provides a natural language interface to [Explainable Boosting Machines (EBMs)](https://github.com/interpretml/interpret). With this package, you can convert the graphs of Explainable Boosting Machines to text and generate prompts for LLMs. We also have higher-level functions that directly ask the LLM to describe entire models. This package is under active development, so the current API is not guaranteed to stay stable.\n\nFeatures:\n- [x] Convert EBMs and their graphs to text that can be understood by LLMs.\n- [x] Ask the LLM to describe and summarize individual graphs or entire models.\n- [x] Modular approach that allows to write custom prompts - ask the LLM to perform any desired task with the EBM.\n- [x] Automatic simplification of minor details in graphs to stay within the desired token limit.\n\n# Installation\n\nPython 3.8+ | Linux, Mac, Windows\n```sh\npip install t2ebm\n```\n\n# High-Level API: Pass the EBM to the LLM\n\nHere, we give an overview of high-level API functions that ask the LLM to describe graphs and EBMs. These functions take care of the prompts for you. The API Reference can be found [here](http://interpret.ml/TalkToEBM/api_reference.html).\n\nFirst, we need to train a model. Here, we train an ```ExplainableBoostingClassifier``` on the Kaggle [Spaceship Titanic Dataset](https://www.kaggle.com/competitions/spaceship-titanic/overview).\n\n```python\nebm = ExplainableBoostingClassifier(feature_names=feature_names)\nebm.fit(X_train, y_train)\n```\nWe can now ask an LLM to describe a graph from the model. Asking the LLM to describe the graph for feature 0, 'Home Planet', is as simple as this:\n\n```python\nimport t2ebm\n\nt2ebm.describe_graph('gpt-4-turbo-2024-04-09', ebm, 0)\n```\n\n\u003e **GPT-4:** *The graph illustrates the effects of the categorical feature `HomePlanet` on a\ndependent variable, as modeled by a Generalized Additive Model (GAM). The\nfeature `HomePlanet` includes three categories: \"Earth,\" \"Europa,\" and \"Mars.\"\nPassengers from Europa exhibit a notably high positive mean effect of 0.5678 on\nthe dependent variable, with a tight confidence interval ranging from 0.5116 to\n0.624, suggesting a strong and consistent positive impact. In contrast, Earth\nshows a negative mean effect of -0.3246 with the confidence interval from -0.354\nto -0.2952, indicating a robust negative association. Mars, while also positive,\nhas a much milder effect of 0.1713, with its confidence interval spanning from\n0.1256 to 0.2171. This pronounced disparity in the effects, especially the\nnegative impact associated with Earth, is surprising and could hint at\nunderlying socio-economic or contextual factors influencing these outcomes.\nUnderstanding these patterns might require further investigation into the\ndataset's characteristics, including possible biases or the nature of the\ndependent variable.*\n\nNote that the LLM does not know about the dataset on which the model was trained. However, we can provide this information. This allows the LLM to talk about the outcome and to describe the graph in the context of the overall dataset. \n\n```python\ndataset_description = \"\"\" The description of the dataset from Kaggle \"\"\"\n\ny_axis_descripton = \"\"\"The y-axis depicts contributions in log-odds towards the probability\nthat the passenger was transported to another dimension.\"\"\"\n```\n\n\n```python\nt2ebm.describe_graph('gpt-4-turbo-2024-04-09', \n                      ebm,\n                      0, \n                      graph_description=y_axis_descripton,\n                      dataset_description=dataset_description)  \n```\n\n\u003e **GPT-4:** *The graph from the Generalized Additive Model (GAM) showcases the contributions\nof different home planets—Earth, Europa, and Mars—towards the log-odds of\npassengers being transported to another dimension on the Spaceship Titanic.\nPassengers from Europa exhibit a strong positive contribution to the likelihood\nof being transported, with a mean log-odds contribution of 0.5678, indicating a\nhigher propensity for these individuals to experience the anomalous event.\nConversely, Earth shows a negative contribution (-0.3246 mean log-odds),\nsuggesting that passengers from Earth are less likely to be transported compared\nto the baseline. Mars, while also showing a positive contribution (0.1713 mean\nlog-odds), has a lesser effect than Europa, but still indicates an increased\nlikelihood over Earth. This pattern could suggest unique environmental,\ntechnological, or cultural factors on Europa and Mars that influence their\ninhabitants' interactions with the spacetime anomaly encountered during the\nvoyage. The observed negative contribution from Earth might hint at more\neffective protective measures or less vulnerability to such spacetime\ndisturbances. These findings highlight planet-based differences in\nsusceptibility to extraordinary events, revealing insights into interplanetary\ntravel and safety in this futuristic scenario.*\n\nInstead of describing individual graphs, we can also ask the LLM to summarize the entire model.\n\n```python\nt2ebm.describe_ebm('gpt-4-turbo-2024-04-09',\n                   ebm,\n                   dataset_description=dataset_description, \n                   y_axis_description=y_axis_descripton) \n```\n\n\u003e **GPT-4:** *The Generalized Additive Model (GAM) used for analyzing the Spaceship Titanic\nanomaly provides crucial insights into factors influencing the likelihood of\npassengers being transported to an alternate dimension. Here’s a concise summary\nof the most impactful features:\n\u003e  1. **CryoSleep**: This feature significantly\naffects the outcome, with passengers in cryosleep more likely to be transported\n(mean effect size 0.814) compared to those who are not (mean effect -0.447).\nThis suggests a unique interaction between the cryosleep state and the anomaly,\npotentially due to the location or conditions of cryosleep chambers.\n\u003e  2. **Spa**: Expenditures on spa services show a strong negative correlation with\nthe likelihood of transportation, particularly at higher spending levels. The\neffect becomes extremely strong (below -5) at the highest expenditures,\nindicating a protective factor potentially linked to socioeconomic status or\nspecific behaviors.\n\u003e  3. **VRDeck**: Similar to spa spending, expenditure on the\nVRDeck is negatively correlated with the probability of transportation,\nintensifying with higher spending. This suggests that engagement in VRDeck\namenities might be associated with safer areas or protective behaviors on the\nship.\n\u003e  4. **RoomService**: Initially, a slight increase in transportation\nlikelihood is observed at very low spending levels on room service, but it\nshifts to a significant negative correlation as spending increases. High\nexpenditures on room service might correlate with safer locations on the ship.\n\u003e  5. **HomePlanet**: Passengers from Europa are much more likely to be transported\n(mean effect 0.5678) compared to those from Earth (mean effect -0.3246) and Mars\n(mean effect 0.1713). This indicates that planetary origin, reflecting differing\nsocio-economic or technological contexts, significantly influences\nsusceptibility to the anomaly.\n\u003e  6. **Cabin**: The cabin location, particularly\ndifferences between Port and Starboard sides, significantly impacts the\nlikelihood of transportation. For instance, Starboard side cabins, especially on\nspecific decks (e.g., \"C/S\" with mean = 2.016), show higher positive effects.\n\u003e  7. **Destination**: The intended destination affects transportation likelihood,\nwith passengers destined for 55 Cancri e exhibiting a higher likelihood compared\nto those heading to PSO J318.5-22 and TRAPPIST-1e. This might be influenced by\nroute or operational parameters specific to each destination.  The model\nhighlights the importance of understanding interactions between passenger states\n(like cryosleep), cabin locations, spending on ship amenities, and origins in\nassessing risks from spacetime anomalies. These factors play crucial roles in\nthe model's predictive accuracy and offer insights for enhancing safety and\ndesign in future interstellar travel scenarios.*\n\n# Low-Level API: Extract Graphs from the EBM and perform custom prompts\n\nMost likely, you want the LLM to perform a custom task with the model. For example, you might want to ask the LLM to find [surprising and counter-intuitive patterns in the graphs](https://arxiv.org/abs/2308.01157). For this, we provide a low-level API that allows us to convert the graphs of EMBs to text. We also provide some basic prompt structures. You should use the functions described here if you want to write your own prompts based on the graphs of EBMs. The API Reference can be found [here](http://interpret.ml/TalkToEBM/api_reference.html).\n\n```python\nimport t2ebm.graphs as graphs\n```\nWe have a simple datastructure for graphs. Here, we extract a graph from an EBM and plot it:\n\n```python\ngraph = graphs.extract_graph(ebm, 9)  # feature 9, 'Spa'\ngraphs.plot_graph(graph)\n```\n\n\u003cimg src=\"images/Spaceship%20Titanic_21_0.png\" alt=\"drawing\" width=\"400\"/\u003e\n\nThe graphs learned by EBMs can contain many small details. We can simplify them to reduce the number of tokens. There is a parameter to control the degree of simplification.\n\n```python\nt2ebm.graphs.plot_graph(t2ebm.graphs.simplify_graph(graph, min_variation_per_cent=0.041))\n```\n\n\u003cimg src=\"images/Spaceship%20Titanic_24_0.png\" alt=\"drawing\" width=\"400\"/\u003e\n\n\nThe package also supports categorical and boolean features\n\n```python\ngraphs.plot_graph(graphs.extract_graph(ebm, 0)) # home planet\n```\n\n\u003cimg src=\"images/Spaceship%20Titanic_22_0.png\" alt=\"drawing\" width=\"400\"/\u003e\n\nBefore we can send a graph to an LLM, we need to convert it to text. The function ```graph_to_text``` takes care of this.\n\n```python\ngraph_as_text = graphs.graph_to_text(graph, max_tokens=1000)\nprint(graph_as_text)\n```\n\n\u003e    This graph represents a continuous-valued feature. The keys are intervals that represent ranges where the function predicts the same value.\n\u003e    \n\u003e    Feature Name: Spa\n\u003e    Feature Type: continuous\n\u003e    Means: {\"(0.0, 36.5)\": 0.517, \"(36.5, 117.5)\": 0.212, ..., \"(3978.0, 18572.0)\": -5.917}\n\u003e \n\u003e    Lower Bounds (95%-Confidence Interval): {\"(0.0, 36.5)\": 0.068, \"(36.5, 117.5)\": 0.049, ..., \"(3978.0, 18572.0)\": -6.956}\n\u003e \u003e \n\u003e    Upper Bounds (95%-Confidence Interval): {\"(0.0, 36.5)\": 0.966, \"(36.5, 117.5)\": 0.374, ..., \"(3978.0, 18572.0)\": -4.877}\n    \nGiven a graph's textual representation, we can embed it in a prompt. The function ```describe_graph``` provides a simple prompt template that can be used as a first step to build more complex prompts. \n\n```python\nt2ebm.prompts.describe_graph(graph_as_text,\n                             graph_description=\"{The y-axis on the graph depicts how cool the graph is.}\",\n                             dataset_description=\"{The dataset is great!}\",\n                             task_description=\"{Please summarize the graph. Pay special attention to run 'pip install t2ebm'}\",)\n```\n\n```\nBelow is the graph of a Generalized Additive Model (GAM). The graph is presented as a JSON object with keys representing the x-axis and values representing the y-axis. For continuous features, the keys are intervals that represent ranges where the function predicts the same value. For categorical features, each key represents a possible value that the feature can take.\n    \nThe graph is provided in the following format:\n    - The name of the feature depicted in the graph\n    - The type of the feature (continuous, categorical, or boolean)\n    - Mean values\n    - Lower bounds of confidence interval (optional)\n    - Upper bounds of confidence interval (optional)\n\nHere is the graph:\n\nThis graph represents categorical feature. Each key represents a possible value that the feature can take.\n\nFeature Name: HomePlanet\nFeature Type: categorical\nMeans: {\"Earth\": -0.3246, \"Europa\": 0.5678, \"Mars\": 0.1713}\nLower Bounds (95%-Confidence Interval): {\"Earth\": -0.354, \"Europa\": 0.5116, \"Mars\": 0.1256}\nUpper Bounds (95%-Confidence Interval): {\"Earth\": -0.2952, \"Europa\": 0.624, \"Mars\": 0.2171}\n\n\n{The y-axis on the graph depicts how cool the graph is.}\n\nHere is a description of the dataset that the model was trained on:\n\n{The dataset is great!}\n\n{Please summarize the graph. Pay special attention to run 'pip install t2ebm'}\n```\n\nWith this, you are ready to write your own prompts and ask the LLM to perform tasks with the graph. \n\n# Citation\n\nIf you use this software in your research, please consider citing our research papers.\n\n```bib\n@inproceedings{bordt2024talktoebm,\n  author    = {Sebastian Bordt, Ben Lengerich, Harsha Nori, Rich Caruana},\n  title     = {Data Science with LLMs and Interpretable Models},\n  booktitle = {XAI4Sci Workshop at AAAI-24},\n  year      = {2024}\n }\n```\n\n```bib\n@inproceedings{lengerich2023llms,\n  author    = {Benjamin J. Lengerich, Sebastian Bordt, Harsha Nori, Mark E. Nunnally, Yin Aphinyanaphongs, Manolis Kellis, and Rich Caruana},\n  title     = {LLMs Understand Glass-Box Models, Discover Surprises, and Suggest Repairs},\n  booktitle = {arxiv},\n  year      = {2023}\n }\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finterpretml%2Ftalktoebm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finterpretml%2Ftalktoebm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finterpretml%2Ftalktoebm/lists"}