{"id":17719829,"url":"https://github.com/zjysteven/mink-plus-plus","last_synced_at":"2025-09-04T10:42:18.913Z","repository":{"id":231246351,"uuid":"781043217","full_name":"zjysteven/mink-plus-plus","owner":"zjysteven","description":"[ICLR'25 Spotlight] Min-K%++: Improved baseline for detecting pre-training data of LLMs","archived":false,"fork":false,"pushed_at":"2025-05-26T17:07:57.000Z","size":3992,"stargazers_count":44,"open_issues_count":0,"forks_count":8,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-01T16:18:38.133Z","etag":null,"topics":["iclr2025","llama","llm","mamba","membership-inference-attack","pretraining-data-detection","pythia"],"latest_commit_sha":null,"homepage":"https://zjysteven.github.io/mink-plus-plus/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zjysteven.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-04-02T16:37:30.000Z","updated_at":"2025-08-31T08:21:47.000Z","dependencies_parsed_at":"2024-06-10T15:46:35.132Z","dependency_job_id":"76f4ed67-7e0f-4d0f-9a14-9075a1185df2","html_url":"https://github.com/zjysteven/mink-plus-plus","commit_stats":{"total_commits":12,"total_committers":2,"mean_commits":6.0,"dds":0.08333333333333337,"last_synced_commit":"f0dbf96ea544edc0edc7393c2634cdcad66d484c"},"previous_names":["zjysteven/mink-plus-plus"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/zjysteven/mink-plus-plus","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjysteven%2Fmink-plus-plus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjysteven%2Fmink-plus-plus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjysteven%2Fmink-plus-plus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjysteven%2Fmink-plus-plus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zjysteven","download_url":"https://codeload.github.com/zjysteven/mink-plus-plus/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjysteven%2Fmink-plus-plus/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273317749,"owners_count":25084037,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-02T02:00:09.530Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["iclr2025","llama","llm","mamba","membership-inference-attack","pretraining-data-detection","pythia"],"created_at":"2024-10-25T15:09:35.020Z","updated_at":"2025-09-04T10:42:18.857Z","avatar_url":"https://github.com/zjysteven.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Min-K%++: Improved Baseline for Detecting Pre-Training Data of LLMs\n\n## Overview\n\n![teaser figure](images/teaser_w_results.png)\n\nWe propose a new Membership Inference Attack method named **Min-K%++** for detecting pre-training data of LLMs, which achieves SOTA results among reference-free methods. This repo contains the lightweight implementation of our method (along with all the baselines) on the [WikiMIA benchmark](https://huggingface.co/datasets/swj0419/WikiMIA). For experiments on the [MIMIR benchmark](https://github.com/iamgroot42/mimir), please refer to our fork [here](https://github.com/zjysteven/mimir).\n\n[![arXiv](https://img.shields.io/badge/arXiv-2404.02936-b31b1b.svg?)](https://arxiv.org/abs/2404.02936)\n[![openreview](https://img.shields.io/badge/ICLR-openreview-blue)](https://openreview.net/forum?id=ZGkfoufDaU)\n\n\n\n## Setup\n### Environment\nFirst install torch according to your environment. Then simply install dependencies by `pip install -r requirements.txt`.\n\nOur code is tested with Python 3.8, PyTorch 2.2.0, Cuda 12.1.\n\n### Data\nAll data splits are hosted on huggingface and will be automatically loaded when running scripts.\n- The original WikiMIA is from [🤗swj0419/WikiMIA](https://huggingface.co/datasets/swj0419/WikiMIA). \n- The WikiMIA authors also studied a *paraphrased* setting, yet the paraphrased data was not released. Here we provide our version, which is paraphrased by ChatGPT with the instruction of replacing certain number of words. The data is hosted at [🤗zjysteven/WikiMIA_paraphrased_perturbed](https://huggingface.co/datasets/zjysteven/WikiMIA_paraphrased_perturbed).\n- In addition, to run Neighbor attack, one needs to perturb each input sentence (with masked language model) to create perturbed neighbors. We also provide the perturbed data for everyone to use at [🤗zjysteven/WikiMIA_paraphrased_perturbed](https://huggingface.co/datasets/zjysteven/WikiMIA_paraphrased_perturbed).\n- Lastly, we propose a new setting that simulates \"detect-while-generating\" by concatenating the training text with the leading non-training text. This split is hosted at [🤗zjysteven/WikiMIA_concat](https://huggingface.co/datasets/zjysteven/WikiMIA_concat).\n\n## Running\nThere are four scripts, each of which is self-contained to best facilitate quick reproduction and extension. The meaning of the arguments of each script should be clear from their naming.\n\n- `run.py` will run the Loss, Zlib, Min-K%, and Min-K%++ attack on the WikiMIA dataset (either the original or the paraphrased version) with the specified model.\n- `run_ref.py` will run the Ref, Lowercase attack on the WikiMIA dataset (either the original or the paraphrased version) with the specified model.\n- `run_neighbor.py` will run the Neighbor attack on the WikiMIA dataset (either the original or the paraphrased version) with the specified model.\n- `run_concat.py` focus on the WikiMIA_concat dataset with the specified model. For this setting only the Loss, Zlib, Min-K%, and Min-K%++ are applicable.\n\nThe outputs of these scripts will be a csv file consisting of method results (AUROC and TPR@FPR=5%) stored in the `results` directory, with the filepath indicating the dataset and model. Sample results by running the four scripts are provided in the `results` directory.\n\n## HF paths of evaluated model in the paper\n- Mamba: [state-spaces/mamba-1.4b-hf](https://huggingface.co/state-spaces/mamba-1.4b-hf)\n, [state-spaces/mamba-2.8b-hf](https://huggingface.co/state-spaces/mamba-2.8b-hf)\n- Pythia: [EleutherAI/pythia-2.8b](https://huggingface.co/EleutherAI/pythia-2.8b), [EleutherAI/pythia-6.9b](https://huggingface.co/EleutherAI/pythia-6.9b), [EleutherAI/pythia-12b](https://huggingface.co/EleutherAI/pythia-12b)\n- GPT-NeoX: [EleutherAI/gpt-neox-20b](https://huggingface.co/EleutherAI/gpt-neox-20b)\n- LLaMA: [huggyllama/llama-13b](https://huggingface.co/huggyllama/llama-13b), [huggyllama/llama-30b](https://huggingface.co/huggyllama/llama-30b), [huggyllama/llama-65b](https://huggingface.co/huggyllama/llama-65b)\n- OPT: [facebook/opt-66b](https://huggingface.co/facebook/opt-66b)\n\n\n## Acknowledgement\nThis codebase is adapted from the [official repo](https://github.com/swj0419/detect-pretrain-code) of Min-K% and WikiMIA.\n\n## Citation\nIf you find this work/repo/data splits useful, please consider citing our paper:\n```\n@inproceedings{\n  zhang2025mink,\n  title={Min-K\\%++: Improved Baseline for Pre-Training Data Detection from Large Language Models},\n  author={Jingyang Zhang and Jingwei Sun and Eric Yeats and Yang Ouyang and Martin Kuo and Jianyi Zhang and Hao Frank Yang and Hai Li},\n  booktitle={The Thirteenth International Conference on Learning Representations},\n  year={2025},\n  url={https://openreview.net/forum?id=ZGkfoufDaU}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzjysteven%2Fmink-plus-plus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzjysteven%2Fmink-plus-plus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzjysteven%2Fmink-plus-plus/lists"}