{"id":19505282,"url":"https://github.com/interpretml/llm-tabular-memorization-checker","last_synced_at":"2025-08-18T11:50:04.119Z","repository":{"id":208632135,"uuid":"718758254","full_name":"interpretml/LLM-Tabular-Memorization-Checker","owner":"interpretml","description":"Testing Language Models for Memorization of Tabular Datasets.","archived":false,"fork":false,"pushed_at":"2025-02-10T12:05:51.000Z","size":18933,"stargazers_count":34,"open_issues_count":0,"forks_count":5,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-08-16T21:48:10.515Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://interpret.ml/LLM-Tabular-Memorization-Checker/api_reference.html","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/interpretml.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-11-14T18:34:51.000Z","updated_at":"2025-07-09T11:39:36.000Z","dependencies_parsed_at":"2024-01-15T20:14:08.725Z","dependency_job_id":"6c958e41-5a57-4daa-a77c-a92e98398f1f","html_url":"https://github.com/interpretml/LLM-Tabular-Memorization-Checker","commit_stats":null,"previous_names":["interpretml/llm-tabular-memorization-checker"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/interpretml/LLM-Tabular-Memorization-Checker","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/interpretml%2FLLM-Tabular-Memorization-Checker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/interpretml%2FLLM-Tabular-Memorization-Checker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/interpretml%2FLLM-Tabular-Memorization-Checker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/interpretml%2FLLM-Tabular-Memorization-Checker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/interpretml","download_url":"https://codeload.github.com/interpretml/LLM-Tabular-Memorization-Checker/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/interpretml%2FLLM-Tabular-Memorization-Checker/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270775856,"owners_count":24642964,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-16T02:00:11.002Z","response_time":91,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T22:29:29.636Z","updated_at":"2025-08-18T11:50:04.091Z","avatar_url":"https://github.com/interpretml.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🐘 Testing Language Models for Memorization of Tabular Datasets\n![PyPI - Version](https://img.shields.io/pypi/v/tabmemcheck)\n![Python](https://img.shields.io/badge/Python-3.9+-blue.svg)\n![License](https://img.shields.io/github/license/interpretml/TalkToEBM.svg?style=flat-square)\n[![tests](https://github.com/interpretml/LLM-Tabular-Memorization-Checker/actions/workflows/run_tests.yaml/badge.svg?branch=main)](https://github.com/interpretml/LLM-Tabular-Memorization-Checker/actions/workflows/run_tests.yaml)\n[![Documentation](https://img.shields.io/badge/Documentation-View-blue)](http://interpret.ml/LLM-Tabular-Memorization-Checker/api_reference.html)\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"img/elephant.webp\" width=\"800\" alt=\"Header Test\"/\u003e\n\u003c/p\u003e\n\nTabmemcheck is an open-source Python library to test language models for memorization of tabular datasets. \n\nThe package provides four different tests for verbatim memorization of a tabular dataset (header test, row completion test, feature completion test, first token test).\n\nIt also provides additional heuristics to test what an LLM knows about a tabular dataset (feature names test, feature values test, dataset name test, and sampling).\n \nFeatures:\n- [x] Test GPT-3.5, GPT-4, and other LLMs for prior exposure with tabular datasets.\n- [x] Supports chat models and (base) language models. In chat mode, we use few-shot learning to condition the model on the desired behavior.\n- [x] The submodule ``tabmemcheck.datasets`` allows to load popular tabular datasets in perturbed form (``original``, ``perturbed``, ``task``, and ``statistical``), as used in our COLM'24 [paper](https://arxiv.org/abs/2404.06209).\n- [x] The [code](https://github.com/interpretml/LLM-Tabular-Memorization-Checker/tree/main/colm-2024-paper-code) to replicate the COLM'24 paper allows to perform few-shot learning with LLMs and tabular data.\n\nThe different memorization tests were first described in a Neurips'23 workshop [paper](https://arxiv.org/abs/2403.06644). \n\nTo see what can be done with this package, take a look at our COLM'24 [paper](https://arxiv.org/abs/2404.06209) *\"Elephants Never Forget: Memorization and Learning of Tabular data in Large Language Models\"*. The code to replicate the results in the paper is [here](https://github.com/interpretml/LLM-Tabular-Memorization-Checker/tree/main/colm-2024-paper-code).\n\nThe API reference is available [here](http://interpret.ml/LLM-Tabular-Memorization-Checker/api_reference.html).\n\nThere are example notebooks for [traditional tabular datasets](https://github.com/interpretml/LLM-Tabular-Memorization-Checker/blob/main/examples/tabular-datasets.ipynb) and datasets used in OpenAI's [MLE-bench](https://github.com/interpretml/LLM-Tabular-Memorization-Checker/blob/main/examples/MLE-bench-contamination.ipynb).\n\n### Installation\n\n```\npip install tabmemcheck\n```\n\nThen use ```import tabmemcheck``` to import the Python package.\n\n# Tests for Verbatim Memorization\n\nThe header test asks the LLM to complete the initial rows of a CSV file.\n\n```python\nheader_prompt, header_completion, response = tabmemcheck.header_test('uci-wine.csv', 'gpt-3.5-turbo-0613', completion_length=350)\n```\n\n\u003cp align=\"left\"\u003e\n  \u003cimg src=\"img/header.png\" width=\"500\" alt=\"Header Test\"/\u003e\n\u003c/p\u003e\n\nHere, we see that ```gpt-3.5-turbo-0613``` can complete the initial rows of the UCI Wine dataset. The function output visualizes the Levenshtein string distance between the actual dataset and the model completion.\n\nThe row completion test asks the LLM to complete random rows of a CSV file.\n\n```python\nrows, responses = tabmemcheck.row_completion_test('iris.csv', 'gpt-4-0125-preview', num_queries=25)\n```\n\n\u003cp align=\"left\"\u003e\n  \u003cimg src=\"img/row_completion.png\" width=\"310\" alt=\"Row Completion Test\"/\u003e\n\u003c/p\u003e\n\nHere, we see that ```gpt-4-0125-preview``` can complete random rows of the Iris dataset. The function output again visualizes the Levenshtein string distance between the actual dataset rows and the model completions.\n\nThe feature completion test asks the LLM to complete the values of a specific feature in the dataset.\n\n```python\nfeature_values, responses = tabmemcheck.feature_completion_test('titanic-train.csv', 'gpt-3.5-turbo-0125', feature_name='Name', num_queries=25)\n```\n\n\u003cp align=\"left\"\u003e\n  \u003cimg src=\"img/feature_completion.png\" width=\"400\" alt=\"Row Completion Test\"/\u003e\n\u003c/p\u003e\n\nHere, we see that ```gpt-3.5-turbo-0125``` can complete the names of the passengers in the Kaggle Titanic dataset. The function output again visualizes the Levenshtein string distance between the feature values in the dataset and the model completions.\n\nThe first token test asks the LLM to complete the first token in the next row of a CSV file.\n\n```python\ntabmemcheck.first_token_test('adult-train.csv', 'gpt-3.5-turbo-0125', num_queries=100)\n```\n\n```\nFirst Token Test: 37/100 exact matches.\nFirst Token Test Baseline (Matches of most common first token): 50/100.\n```\nHere, the test provides no evidence of memorization of the Adult Income dataset in ```gpt-3.5-turbo-0125```.\n\nOne of the key features of this package is that we have implemented prompts that allow us to run the various completion tests not only with (base) language models but also with chat models (specifically, GPT-3.5 and GPT-4).\n\nThere is also a simple way to run all the different tests and generate a small report.\n\n```python\ntabmemcheck.run_all_tests(\"adult-test.csv\", \"gpt-4-0613\")\n```\n\n# Other contamination tests\n\nThe feature names test asks the LLM to complete the feature names of a dataset.\n\n```python\ntabmemcheck.feature_names_test('Kaggle Tabular Playground Series Dec 2021.csv.csv', 'gpt-4o-2024-08-06')\n```\n\n\u003cp align=\"left\"\u003e\n  \u003cimg src=\"img/feature_names.png\" width=\"500\" alt=\"Header Test\"/\u003e\n\u003c/p\u003e\n\nThe feature values test asks the LLM to provide a typical observation from the dataset.\n\n```python\ntabmemcheck.feature_values_test('OSIC Pulmonary Fibrosis Progression.csv', 'gpt-4o-2024-08-06')\n```\n\n\u003cp align=\"left\"\u003e\n  \u003cimg src=\"img/feature_values.png\" width=\"500\" alt=\"Header Test\"/\u003e\n\u003c/p\u003e\n\nMore generally, you can use ```sample``` to ask the LLM to provide samples from the dataset.\n\n```python\ntabmemcheck.sample('OSIC Pulmonary Fibrosis Progression.csv', 'gpt-4o-2024-08-06')\n```\n\n\u003cp align=\"left\"\u003e\n  \u003cimg src=\"img/samples.png\" width=\"500\" alt=\"Header Test\"/\u003e\n\u003c/p\u003e\n\n\nThe dataset name test asks the LLM to provide the name of the dataset, given the initial rows of the CSV file.\n\n```python\ntabmemcheck.dataset_name_test('spooky author identification train.csv', 'gpt-4o-2024-08-06')\n```\n\n\u003cp align=\"left\"\u003e\n  \u003cimg src=\"img/dataset_name.png\" width=\"500\" alt=\"Header Test\"/\u003e\n\u003c/p\u003e\n\n\n# How should the results of the tests be interpreted?\n\nWe have often been asked how the results of the different tests should be interpreted. For example, do 3 out of 25 correctly completed rows in the row completion test mean the dataset is memorized?  The key point in interpreting the test results is that one has to consider the amount of entropy in the dataset. \n\nAt a high level, we want to say that a dataset is memorized if an LLM can consistently generate it. However, this only makes sense if the dataset is not a (deterministic) string sequence that can simply be predicted by the LLM. In most tabular datasets, we don't have to worry about this too much. This is because they contain random variables, and it is impossible to consistently reproduce the realizations of random variables unless the values of the random variables have been seen before (that is, during training).\n\nWhen we judge the test results, we have to consider the completion rate of the LLM and the amount of entropy in the dataset. For example, the OpenML Diabetes dataset contains an individual's glucose level, blood pressure, and BMI, as well as other measurements that are at least in part random. Now, if an LLM can consistently generate even a few rows of this unique dataset, this is fairly strong evidence of memorization (see [Carlini et al. 2019](https://arxiv.org/abs/1802.08232) and [Carlini et al. 2021](https://arxiv.org/abs/2012.07805) if you are interested in details). To give a contrary example, the Iris dataset contains many rows that are near-duplicates. This means that an LLM might also achieve a non-zero row completion rate by chance or prediction, and one could not conclude that the dataset was seen during pre-training from the fact that an LLM can generate a few rows.\n\nBecause one needs to weight the completions of the LLM against the entropy in the dataset, it is unfortunately impossible to give a general ratio such as \"X out of 100 completed rows imply memorization\". \n\nWhile this all sounds very complex, the practical evidence for memorization is often very clear. This can also be seen in the examples above.\n\n# How do the tests work?\n\nWe use few-shot learning to condition chat models on the desired task. This works well for GPT-3.5 and GPT-4, and also for many other LLMs (but not necessarily for all LLMs). \n\nYou can set ```tabmemcheck.config.print_prompts = True``` to see the prompts.\n\nYou can set ```tabmemcheck.config.print_responses = True``` to print the LLM responses, a useful sanity check.\n\n# Can I use this package to write my own tests?\n\nYes. The module [chat_completion.py](https://github.com/interpretml/LLM-Tabular-Memorization-Checker/blob/main/tabmemcheck/chat_completion.py) provides the general-purpose function ```prefix_suffix_chat_completion``` which is used to implement most of the different tests. \n\nYou can see how ```prefix_suffix_chat_completion``` is being used by reading the implementations of the different tests in [functions.py](https://github.com/interpretml/LLM-Tabular-Memorization-Checker/blob/main/tabmemcheck/functions.py).\n\nWe also provide the general-purpose function ```chat_completion```, which again relies on ```prefix_suffix_chat_completion```.\n\n# Using the package with your own LLM\n\nTo test your own LLM, simply implement ```tabmemcheck.LLM_Interface```. We use the OpenAI message format.\n\n```python\n@dataclass\nclass LLM_Interface:\n    \"\"\"Generic interface to a language model.\"\"\"\n\n    # if true, the tests use the chat_completion function, otherwise the completion function\n    chat_mode = False\n\n    def completion(self, prompt: str, temperature: float, max_tokens: int):\n        \"\"\"Send a query to a language model.\n\n        :param prompt: The prompt (string) to send to the model.\n        :param temperature: The sampling temperature.\n        :param max_tokens: The maximum number of tokens to generate.\n\n        Returns:\n            str: The model response.\n        \"\"\"\n        raise NotImplementedError\n\n    def chat_completion(self, messages, temperature: float, max_tokens: int):\n        \"\"\"Send a query to a chat model.\n\n        :param messages: The messages to send to the model. We use the OpenAI format.\n        :param temperature: The sampling temperature.\n        :param max_tokens: The maximum number of tokens to generate.\n\n        Returns:\n            str: The model response.\n        \"\"\"\n        raise NotImplementedError\n```\n\n# Limitations\n\nThe tests provided in this package do not guarantee that the LLM has **not** seen or memorized the data. Specifically, it might not be possible to extract the data from the LLM via prompting, even though the LLM has memorized it.\n\n\u003c!--- # FAQ\n\n- Q: What is the relationship between this tool and other techniques such as membership inference queries?\n\nA: Our main objective is to provide tests that can be performed with the kind of query access provided to LLMs like GPT-4. This means that we assume no access to the probability distribution over tokens ('logprobs') and might only be able to run a limited number of queries. In general, test that rely on the logprobs, or even the ability to re-train the model, could be more powerful than our tests. \n\n- Q: What are the prompts that you use for the different tests?\n\nA: The prompts are given in the Supplement of our [paper](https://openreview.net/pdf?id=l1u7jA60wT). You can also set\n\n```python\ntabmemcheck.config.print_prompts = True\ntabmemcheck.config.print_responses = True\n```\n\nor use the option ```--debug``` with the CLI tool to see the specific prompts and responses for your queries. \n\n- How can I replicate the results in the paper?\n\n  A: the code will come soon.\n\n---\u003e\n\n# Citation\n\nIf you find this code useful in your research, please consider citing our research papers.\n\n```\n@inproceedings{bordt2024colm,\n  title={Elephants Never Forget: Memorization and Learning of Tabular Data in\n  Large Language Models},\n  author={Bordt, Sebastian and Nori, Harsha and Rodrigues, Vanessa and Nushi, Besmira and Caruana, Rich},\n  booktitle={Conference on Language Modeling (COLM)},\n  year={2024}\n}\n\n@inproceedings{bordt2023testing,\n  title={Elephants Never Forget: Testing Language Models for Memorization of Tabular Data},\n  author={Bordt, Sebastian and Nori, Harsha and Caruana, Rich},\n  booktitle={NeurIPS 2023 Second Table Representation Learning Workshop},\n  year={2023}\n}\n```\n\n# References\n\nChang et al., [\"Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4\"](https://arxiv.org/abs/2305.00118), EMNLP 2023\n\nCarlini et al., [\"Extracting Training Data from Large Language Models\"](https://arxiv.org/abs/2012.07805), USENIX Security Symposium 2021\n\nCarlini et al., [\"The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks\"](https://arxiv.org/abs/1802.08232), USENIX Security Symposium 2019 \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finterpretml%2Fllm-tabular-memorization-checker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finterpretml%2Fllm-tabular-memorization-checker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finterpretml%2Fllm-tabular-memorization-checker/lists"}