https://github.com/SALT-NLP/Efficient_Unlearning

https://github.com/SALT-NLP/Efficient_Unlearning

Last synced: 2 months ago
JSON representation

• Host: GitHub
• URL: https://github.com/SALT-NLP/Efficient_Unlearning
• Owner: SALT-NLP
• License: apache-2.0
• Created: 2023-10-18T18:05:54.000Z (12 months ago)
• Default Branch: main
• Last Pushed: 2023-10-18T22:53:03.000Z (12 months ago)
• Last Synced: 2024-07-26T05:06:04.658Z (2 months ago)
• Language: Python
• Size: 3.62 MB
• Stars: 29
• Watchers: 1
• Forks: 0
• Open Issues: 5
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-llm-unlearning - GitHub

README

        

# 📜 Efficient_Unlearning

🔍 This is the official code and data repository for the EMNLP 2023 paper titled "Unlearn What You Want to Forget: Efficient Unlearning for LLMs".

* The codes will be updated soon.

## 🌟 Abstract

Abstract

Large language models (LLMs) have achieved significant progress from pre-training on and memorizing a wide range of textual data, however, this process might suffer from privacy issues and violations of data protection regulations. As a result, the ability to easily remove data related to individual users from such models while not deteriorating their predictive quality after the removal becomes increasingly important. To address these issues, in this work, we propose an efficient unlearning framework that could efficiently update LLMs without having to retrain the whole model after data removals, by introducing lightweight unlearning layers learned with a selective teacher-student objective into the transformers. In addition, we introduce a fusion mechanism to effectively combine different unlearning layers that learns to forget different sets of data to handle a sequence of forgetting operations. Experiments on classification and generation tasks demonstrate the effectiveness of our proposed methods compared to the state-of-the-art baselines.

## 📊 Data Description

📥 **Download Data**: [Download Link]

## 🚀 How to Run

### Environment Setup

```

conda install mpi4py

conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch

pip install -e /src/models/transformers/

pip install spacy==3.2.4

pip install datasets==1.8.0 

pip install huggingface_hub==0.4.0 

pip install wandb

```

### 🧠 Single Unlearn

```

python src/run.py --unlearn --forgot_data "forgot.csv" --retained_data "retained.csv"

```

### 🌐 Fusing Unlearning Layers

```

python src/run.py --merge --merge_set "the folder of the weights to merge"

```