{"id":13458935,"url":"https://github.com/chrisliu298/llm-unlearn-eco","last_synced_at":"2025-05-16T00:18:53.484Z","repository":{"id":246463796,"uuid":"821214730","full_name":"chrisliu298/llm-unlearn-eco","owner":"chrisliu298","description":"[NeurIPS 2024] Large Language Model Unlearning via Embedding-Corrupted Prompts","archived":false,"fork":false,"pushed_at":"2024-09-26T04:09:15.000Z","size":423,"stargazers_count":25,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-24T16:38:09.444Z","etag":null,"topics":["alignment","large-language-models","llm-unlearning","machine-unlearning","unlearning"],"latest_commit_sha":null,"homepage":"https://chrisliu298.ai/llm-unlearn-eco-project-page/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chrisliu298.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-28T04:10:07.000Z","updated_at":"2025-02-26T05:06:09.000Z","dependencies_parsed_at":"2024-06-28T04:45:21.271Z","dependency_job_id":"cde78541-0fc8-47c8-8549-6a3d88a364c9","html_url":"https://github.com/chrisliu298/llm-unlearn-eco","commit_stats":null,"previous_names":["chrisliu298/llm-unlearn-eco"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisliu298%2Fllm-unlearn-eco","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisliu298%2Fllm-unlearn-eco/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisliu298%2Fllm-unlearn-eco/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisliu298%2Fllm-unlearn-eco/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chrisliu298","download_url":"https://codeload.github.com/chrisliu298/llm-unlearn-eco/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254442811,"owners_count":22071889,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["alignment","large-language-models","llm-unlearning","machine-unlearning","unlearning"],"created_at":"2024-07-31T09:00:59.958Z","updated_at":"2025-05-16T00:18:53.460Z","avatar_url":"https://github.com/chrisliu298.png","language":"Python","funding_links":[],"categories":["Papers"],"sub_categories":["2024"],"readme":"# [NeurIPS 2024] Large Language Model Unlearning via Embedding-Corrupted Prompts\n\nThis repository contains the code of our paper [Large Language Model Unlearning via Embedding-Corrupted Prompts](https://arxiv.org/abs/2406.07933). In this paper, we propose an extremely lightweight LLM unlearning scheme for powerful non-open-weight LLMs, approaching LLM unlearning from the input perspective.\n\nOur project page is available [here](https://chrisliu298.github.io/llm-unlearn-eco-project-page/).\n\n![ECO diagram](assets/eco.png)\n\n## Table of Contents\n\n- [Table of Contents](#table-of-contents)\n- [Abstract](#abstract)\n- [Code Structure](#code-structure)\n- [Installing Requirements](#installing-requirements)\n- [Demo](#demo)\n- [Experiments](#experiments)\n  - [Prompt Classifiers](#prompt-classifiers)\n  - [Entity Unlearning](#entity-unlearning)\n  - [Hazardous Knowledge Unlearning](#hazardous-knowledge-unlearning)\n  - [Copyrighted Content Unlearning](#copyrighted-content-unlearning)\n  - [Other Experiments](#other-experiments)\n- [Citation](#citation)\n\n## Abstract\n\nLarge language models (LLMs) have advanced to encompass extensive knowledge across diverse domains. Yet controlling what a large language model should not know is important for ensuring alignment and thus safe use. However, accurately and efficiently unlearning knowledge from an LLM remains challenging due to the potential collateral damage caused by the fuzzy boundary between retention and forgetting, and the large computational requirements for optimization across state-of-the-art models with hundreds of billions of parameters. In this work, we present Embedding-COrrupted (ECO) Prompts, a lightweight unlearning framework for large language models to address both the challenges of knowledge entanglement and unlearning efficiency. Instead of relying on the LLM itself to unlearn, we enforce an unlearned state during inference by employing a prompt classifier to identify and safeguard prompts to forget. We learn corruptions added to prompt embeddings via zeroth order optimization toward the unlearning objective offline and corrupt prompts flagged by the classifier during inference. We find that these embedding-corrupted prompts not only lead to desirable outputs that satisfy the unlearning objective but also closely approximate the output from a model that has never been trained on the data intended for forgetting. Through extensive experiments on unlearning, we demonstrate the superiority of our method in achieving promising unlearning at nearly zero side effects in general domains and domains closely related to the unlearned ones. Additionally, we highlight the scalability of our method to 100 LLMs, ranging from 0.5B to 236B parameters, incurring no additional cost as the number of parameters increases.\n\n## Code Structure\n\n- `eco/attack`: Classifiers, corruption functions, a corrupted model wrapper, and utility functions.\n- `eco/dataset`: All datasets used in the experiments.\n- `eco/evaluator`: Evaluation metrics and utility functions.\n- `eco/model`: The base model wrapper.\n- `eco/optimizer`: The zeroth order optimization module.\n- `eco/inference.py`: The main inference script, supporting both logit-based evaluation and generation.\n\n## Installing Requirements\n\nTo install the required packages, run\n\n```shell\npython -m pip install -r requirements.txt\npython -m pip install --no-cache-dir flash-attn --no-build-isolation\npython -m spacy download en_core_web_sm\nexport HF_HUB_ENABLE_HF_TRANSFER=1\n```\n\n## Demo\n\n`demo.py` contains a minimal example of how to use the ECO method. Given the prompt `Who is Harry Potter?`, we corrupt the prompt by adding random noise to the first dimension of the `Harry` and `Potter` tokens. The corrupted prompt is then fed to the LLM to generate a response using greedy decoding.\n\nUnder the hood, our implementation wraps a HF model and has its own `forward` and `generate` functions. The prompt classification step and the corruption step are executed before the forward pass.\n\n```python\nmodel = ...\nprompt = \"Who is Harry Potter?\"\n# Apply the corruption pattern as a forward hook, which executes the corruption\n# function before the forward pass\napply_corruption_hook(\n    get_nested_attr(model.model, model.model_config[\"attack_module\"]),\n    corrupt_method=\"rand_noise_first_n\",\n    # Corrupt \"Harry\" and \"Potter\" tokens\n    corrupt_args={\n        \"pos\": [[0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0]],\n        \"dims\": 1,\n        \"strength\": 100,\n    },\n)\n# Generate just like a normal model\ngenerated = model.generate(\n    **model.tokenizer(prompt, add_special_tokens=False, return_tensors=\"pt\").to(\n        model.device\n    ),\n    generation_config=model.generation_config,\n    eos_token_id=model.tokenizer.eos_token_id,\n)\n# Remove the hook so that it's not applied to future generations\nremove_hooks(model.model)\nprint(model.tokenizer.batch_decode(generated, skip_special_tokens=False)[0][len(prompt) :])\n# Output:\n# I'm just an AI, I don't have a personal identity or a physical presence.\n# I exist solely as a digital entity, designed to provide information and assist with tasks to the best of my abilities.\n# I don't have personal experiences, emotions, or consciousness like humans do.\n# I'm here to help answer your questions and provide assistance, so feel free to ask me anything!\u003c|eot_id|\u003e\n```\n\nIn the above output, `Meta-Llama-3-8B-Instruct` generates a template-like response that rejects the query.\n\n## Experiments\n\n### Prompt Classifiers\n\nBefore unlearning, we first train a classifier for each dataset/task. The classifier is used during inference: it produces a signal of whether the tokens in the prompt should be corrupted, before the prompt is fed to the LLM. To train a prompt classifier (e.g., for WMDP), run\n\n```shell\npython -m scripts.train_classifier --dataset_name wmdp\n```\n\nAll classifiers used in the paper are also available on Hugging Face Model Hub:\n\n- [tofu_forget01_classifier](https://huggingface.co/chrisliu298/tofu_forget01_classifier)\n- [tofu_forget05_classifier](https://huggingface.co/chrisliu298/tofu_forget05_classifier)\n- [tofu_forget10_classifier](https://huggingface.co/chrisliu298/tofu_forget10_classifier)\n- [wmdp_classifier](https://huggingface.co/chrisliu298/wmdp_classifier)\n- [synthetic_wmdp_classifier](https://huggingface.co/chrisliu298/synthetic_wmdp_classifier)\n- [mmlu-economics_classifier](https://huggingface.co/chrisliu298/mmlu-economics_classifier)\n- [mmlu-law_classifier](https://huggingface.co/chrisliu298/mmlu-law_classifier)\n- [mmlu-physics_classifier](https://huggingface.co/chrisliu298/mmlu-physics_classifier)\n- [bbc_news_classifier](https://huggingface.co/chrisliu298/bbc_news_classifier)\n- [hp_book_classifier](https://huggingface.co/chrisliu298/hp_book_classifier)\n\nHowever, you might need to store the classifiers locally for the experiments listed below. Please make sure to place the classifiers in the corresponding directories specified in the instructions below.\n\n### Entity Unlearning\n\nThe following commands are tested on 2x A100 GPUs (80 GB).\n\nTo fine-tune models subject to unlearning and use the baseline unlearning methods, see the official [TOFU repositoy](https://github.com/locuslab/tofu). For the NPO baselines, we use their [official implementation](https://github.com/licong-lin/negative-preference-optimization).\n\nWe consider two models, `phi-1_5` and `Llama2-7b-chat-hf`, for this task. Before running the scripts below, make sure to place the corresponding model (subject to unlearning) in the current directory. For example, to run unlearning for `Llama2-7b-chat-hf` on the `forget01` subset, you should store the model in `./Llama-2-7b-chat-hf_forget01/`. The `_full` models are trained on the entire dataset, while the `_forgetXX` models are trained on the dataset with the corresponding percentage of entities removed.\n\n```text\n./Llama-2-7b-chat-hf_forget01\n./Llama-2-7b-chat-hf_forget05\n./Llama-2-7b-chat-hf_forget10\n./Llama-2-7b-chat-hf_full\n./phi-1_5_forget01\n./phi-1_5_forget05\n./phi-1_5_forget10\n./phi-1_5_full\n```\n\nThe prompt classifiers should have the following structure:\n\n```text\n./tofu_classifiers/forget01\n./tofu_classifiers/forget05\n./tofu_classifiers/forget10\n```\n\nFor this particular task, we also use token classifiers to selectively corrupt name tokens in the prompts. The token classifier is simply [an NER model](https://huggingface.co/dslim/bert-base-NER) and will be loaded automatically when running the scripts.\n\nThe following commands executes the methods defined in configuration `config/task_config/tofu.yaml`. You may add other configurations to the `config/task_config` directory and run the corresponding scripts. The `--optimal_corrupt_dim` below is for the `zero_out_top_k` corruption method.\n\n```shell\nmodel=phi-1_5\npython -m scripts.evaluate_tofu --forget_set_name forget01 --model_name ${model} --batch_size 64 --classifier_threshold 0.99 --task_config config/task_config/tofu.yaml --optimal_corrupt_dim 507\npython -m scripts.evaluate_tofu --forget_set_name forget05 --model_name ${model} --batch_size 64 --classifier_threshold 0.99 --task_config config/task_config/tofu.yaml --optimal_corrupt_dim 766\npython -m scripts.evaluate_tofu --forget_set_name forget10 --model_name ${model} --batch_size 64 --classifier_threshold 0.9999 --task_config config/task_config/tofu.yaml --optimal_corrupt_dim 1243\n\nmodel=Llama2-7b-chat-hf\npython -m scripts.evaluate_tofu --forget_set_name forget01 --model_name ${model} --batch_size 64 --classifier_threshold 0.99 --task_config config/task_config/tofu.yaml --optimal_corrupt_dim 670\npython -m scripts.evaluate_tofu --forget_set_name forget05 --model_name ${model} --batch_size 64 --classifier_threshold 0.99 --task_config config/task_config/tofu.yaml --optimal_corrupt_dim 881\npython -m scripts.evaluate_tofu --forget_set_name forget10 --model_name ${model} --batch_size 64 --classifier_threshold 0.9999 --task_config config/task_config/tofu.yaml --optimal_corrupt_dim 1000\n```\n\nFor ablation experiments using other corruption methods and parameters, we use the `config/task_config/tofu_ablation_${model}.yaml` configuration files. The following commands evaluate the ablation experiments for the `phi-1_5` and `Llama-2-7b-chat-hf` models:\n\n```shell\nmodel=phi-1_5\npython evaluate_tofu.py --forget_set_name forget01 --model_name phi-1_5 --batch_size 64 --classifier_threshold 0.99 --task_config \"config/task_config/tofu_ablation_${model}.yaml\"\npython evaluate_tofu.py --forget_set_name forget05 --model_name phi-1_5 --batch_size 64 --classifier_threshold 0.99 --task_config \"config/task_config/tofu_ablation_${model}.yaml\"\npython evaluate_tofu.py --forget_set_name forget10 --model_name phi-1_5 --batch_size 64 --classifier_threshold 0.9999 --task_config \"config/task_config/tofu_ablation_${model}.yaml\"\n\nmodel=Llama-2-7b-chat-hf\npython evaluate_tofu.py --forget_set_name forget01 --model_name Llama-2-7b-chat-hf --batch_size 64 --classifier_threshold 0.99 --task_config \"config/task_config/tofu_ablation_${model}.yaml\"\npython evaluate_tofu.py --forget_set_name forget05 --model_name Llama-2-7b-chat-hf --batch_size 64 --classifier_threshold 0.99 --task_config \"config/task_config/tofu_ablation_${model}.yaml\"\npython evaluate_tofu.py --forget_set_name forget10 --model_name Llama-2-7b-chat-hf --batch_size 64 --classifier_threshold 0.9999 --task_config \"config/task_config/tofu_ablation_${model}.yaml\"\n```\n\n### Hazardous Knowledge Unlearning\n\nThe following commands are tested on 2x A100 GPUs (80 GB). For larger models (e.g., with more than 200B parameters), we recommand using at least 3x A100 GPUs (80 GB).\n\nBelow we include commands for evaluating the original model, the unlearned model, and the prompting baseline for the hazardous knowledge unlearning task using the WMDP dataset. The WMDP promopt classifiers should be placed in the `./wmdp_classifier` directory.\n\nTo evaluate unlearning (e.g., with the `Meta-Llama-3-8B-Instruct` model), we use the following commands:\n\n```shell\n# WMDP\nseed=0\nbatch_size=4\nmodel=Meta-Llama-3-8B-Instruct\n# Evaluate original model\npython -m scripts.evaluate_wmdp --model_name $model --batch_size $batch_size --task_config config/task_config/multiple_choice_original.yaml --classifier_threshold 0.999\n# Unlearn using 1-dimension random noise and evaluate\npython -m scripts.evaluate_wmdp --model_name $model --batch_size $batch_size --task_config config/task_config/multiple_choice_rand_noise_1.yaml --classifier_threshold 0.999 --seed ${seed}\n# Evaluate prompting baseline\npython -m scripts.evaluate_wmdp --model_name $model --batch_size $batch_size --task_config config/task_config/multiple_choice_original.yaml --classifier_threshold 0.999 --use_prefix\n\n# MMLU subsets\nsubset=economics\nif [[ \"$subset\" == \"economics\" || \"$subset\" == \"physics\" ]]; then\n    threshold=0.999\nelif [[ \"$subset\" == \"law\" ]]; then\n    threshold=0.995\nfi\n# Evaluate original model\npython -m scripts.evaluate_mmlu_subset --subset_name $subset --model_name $model --batch_size $batch_size --task_config config/task_config/multiple_choice_original.yaml --classifier_threshold ${threshold} --mmlu_subset_only\n# Unlearn using 1-dimension random noise and evaluate\npython -m scripts.evaluate_mmlu_subset --subset_name $subset --model_name $model --batch_size $batch_size --task_config config/task_config/multiple_choice_rand_noise_1.yaml --classifier_threshold ${threshold} --seed ${seed} --mmlu_subset_only\n# Evaluate prompting baseline\npython -m scripts.evaluate_mmlu_subset --subset_name $subset --model_name $model --batch_size $batch_size --task_config config/task_config/multiple_choice_original.yaml --classifier_threshold ${threshold} --mmlu_subset_only --use_prefix\n```\n\n### Copyrighted Content Unlearning\n\nThe following commands are tested on 2x A100 GPUs (80 GB). The `OLMo-1.7-7B-hf` model is used for this task. The `bbc_news` dataset is used for this task. Due to copyright restrictions, we cannot provide the HP Book dataset. The prompt classifiers should be placed in the `./hp_book_classifier` and `./bbc_news_classifier` directories.\n\nTo obtain the models subject to unlearning, we first fine-tune the base models on the corresponding copyrighted content dataset:\n\n```shell\ndataset=bbc_news\nmodel=allenai/OLMo-1.7-7B-hf\naccelerate launch \\\n    --config_file config/training_config/accelerate_config.yaml \\\n    --use_deepspeed \\\n    --deepspeed_config_file config/training_config/deepspeed_config.json \\\n    scripts/finetune_copyright_lm.py \\\n    --dataset_name ${dataset} \\\n    --model_name ${model} \\\n    --num_epochs 5 \\\n    --lr 2e-5 \\\n    --batch_size 4\n```\n\nFor all unlearning baselines, run the following commands:\n\n```shell\nmodel=OLMo-1.7-7B-hf\n# Fine-tune\naccelerate launch \\\n  --config_file config/training_config/accelerate_config.yaml \\\n  --use_deepspeed \\\n  --deepspeed_config_file config/training_config/deepspeed_config.json \\\n  scripts/copyright_unlearn_baselines.py \\\n  --dataset_name ${dataset} \\\n  --model_name ${model} \\\n  --model_path ${dataset}_${model} \\\n  --num_epochs 5 \\\n  --lr 2e-5 \\\n  --batch_size 4 \\\n  --loss ft\n# Gradient ascent\naccelerate launch \\\n  --config_file config/training_config/accelerate_config.yaml \\\n  --use_deepspeed \\\n  --deepspeed_config_file config/training_config/deepspeed_config.json \\\n  scripts/copyright_unlearn_baselines.py \\\n  --dataset_name ${dataset} \\\n  --model_name ${model} \\\n  --model_path ${dataset}_${model} \\\n  --num_epochs 5 \\\n  --lr 2e-5 \\\n  --batch_size 4 \\\n  --loss ga\n# Gradient difference\naccelerate launch \\\n  --config_file config/training_config/accelerate_config.yaml \\\n  --use_deepspeed \\\n  --deepspeed_config_file config/training_config/deepspeed_config.json \\\n  scripts/copyright_unlearn_baselines.py \\\n  --dataset_name ${dataset} \\\n  --model_name ${model} \\\n  --model_path ${dataset}_${model} \\\n  --num_epochs 5 \\\n  --lr 2e-5 \\\n  --batch_size 4 \\\n  --loss gd\n# KL minimization\naccelerate launch \\\n  --config_file config/training_config/accelerate_config.yaml \\\n  --use_deepspeed \\\n  --deepspeed_config_file config/training_config/deepspeed_config.json \\\n  scripts/copyright_unlearn_baselines.py \\\n  --dataset_name ${dataset} \\\n  --model_name ${model} \\\n  --model_path ${dataset}_${model} \\\n  --num_epochs 5 \\\n  --lr 2e-5 \\\n  --batch_size 4 \\\n  --loss kl\n# Mismatch\naccelerate launch \\\n  --config_file config/training_config/accelerate_config.yaml \\\n  --use_deepspeed \\\n  --deepspeed_config_file config/training_config/deepspeed_config.json \\\n  scripts/copyright_unlearn_baselines.py \\\n  --dataset_name ${dataset} \\\n  --model_name ${model} \\\n  --model_path ${dataset}_${model} \\\n  --num_epochs 5 \\\n  --lr 2e-5 \\\n  --batch_size 4 \\\n  --loss rd\n# LLMU\naccelerate launch \\\n  --config_file config/training_config/accelerate_config.yaml \\\n  --use_deepspeed \\\n  --deepspeed_config_file config/training_config/deepspeed_config.json \\\n  scripts/copyright_unlearn_baselines.py \\\n  --dataset_name ${dataset} \\\n  --model_name ${model} \\\n  --model_path ${dataset}_${model} \\\n  --num_epochs 5 \\\n  --lr 2e-5 \\\n  --batch_size 4 \\\n  --loss llmu \\\n  --epsilon1 0.1 \\\n  --epsilon2 1 \\\n  --epsilon3 1\n# SCRUB\naccelerate launch \\\n  --config_file config/training_config/accelerate_config.yaml \\\n  --use_deepspeed \\\n  --deepspeed_config_file config/training_config/deepspeed_config.json \\\n  scripts/copyright_unlearn_baselines.py \\\n  --dataset_name ${dataset} \\\n  --model_name ${model} \\\n  --model_path ${dataset}_${model} \\\n  --num_epochs 5 \\\n  --lr 2e-5 \\\n  --batch_size 4 \\\n  --loss scrub \\\n  --epsilon1 0.001 \\\n  --epsilon2 1\n```\n\nTo evaluate unlearning for all models, make sure all models are placed in the current directory and run the following commands:\n\n```shell\ndataset=bbc_news\nmodel=OLMo-1.7-7B-hf\nbatch_size=4\n# Evaluate forgetting\npython -m scripts.evaluate_copyright --dataset_name ${dataset} --model_name ${model} --batch_size ${batch_size}\n# Evaluate general performance\npython -m scripts.evaluate_general_local --dataset_name ${dataset} --model_name ${model} --batch_size ${batch_size}\n```\n\n### Other Experiments\n\n- `scripts/zeroth_order_optim.py` implements the zeroth order optimization for the corruption strength parameter on the WMDP dataset.\n- `scripts/repeat_test_and_qa.py` studies LLMs response behavior on repeating the corrupted prompts. This corresponds to Appendix E.6 in the paper.\n- `scripts/linear_probe.py` studies if the correct answer of the WMDP questions can be recovered by probing the logit vectors. Note that the logit vectors need to be prepared before running this script.\n\n## Citation\n\n```bibtex\n@article{liu2024large,\n  title={Large Language Model Unlearning via Embedding-Corrupted Prompts},\n  author={Liu, Chris Yuhao and Wang, Yaxuan and Flanigan, Jeffrey and Liu, Yang},\n  journal={arXiv preprint arXiv:2406.07933},\n  year={2024}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchrisliu298%2Fllm-unlearn-eco","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchrisliu298%2Fllm-unlearn-eco","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchrisliu298%2Fllm-unlearn-eco/lists"}