https://github.com/timoschick/self-debiasing
This repository contains the code for "Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP".
https://github.com/timoschick/self-debiasing
Last synced: about 1 year ago
JSON representation
This repository contains the code for "Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP".
- Host: GitHub
- URL: https://github.com/timoschick/self-debiasing
- Owner: timoschick
- License: apache-2.0
- Created: 2021-03-02T15:04:35.000Z (over 5 years ago)
- Default Branch: main
- Last Pushed: 2021-08-20T16:53:56.000Z (almost 5 years ago)
- Last Synced: 2025-04-14T03:09:05.342Z (about 1 year ago)
- Language: Python
- Homepage: https://arxiv.org/abs/2103.00453
- Size: 23.4 KB
- Stars: 88
- Watchers: 1
- Forks: 26
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Self-Diagnosis and Self-Debiasing
This repository contains the source code for [Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP](https://arxiv.org/pdf/2103.00453.pdf).
## 🔧 Setup
All requirements can be found in ``requirements.txt``. You can install all required packages with ``pip install -r requirements.txt``.
## 🩺 Self-Diagnosis
To reproduce the self-diagnosis experiments in the paper, run ``python3 self_diagnosis.py --examples_filename --output_filename `` where
- ```` is the path to a JSONL file containing the texts to self-diagnose (with gold labels). Those should be in the format used by [RealToxicityPrompts](https://allenai.org/data/real-toxicity-prompts). Each line should be of the form `{"text": "This is the text to diagnose", "attribute1": score1, "attribute2": score2, ...}`, for example:
```javascript
{"text": "That Bernie Sanders doesn't want to add over and over again to the list of amendments to be considered", "insult": 0.10827508, "flirtation": 0.25115338, "toxicity": 0.11353362, "profanity": 0.06803029, "sexually_explicit": 0.08224664, "severe_toxicity": 0.04441638, "threat": 0.17986308, "identity_attack": 0.14791483}
```
- ```` is the path to a plain text file to which all outputs are written.
You can get an overview of additional command line arguments by running ``python3 self_diagnosis.py --help``.
## 🩹 Self-Debiasing
To reproduce the self-debiasing experiments in the paper, run ``python3 self_debiasing.py --prompts_filename --output_dir --api_key `` where
- ```` is the path to a JSONL file containing the prompts for which continuations should be predicted. Those should be in the format used by [RealToxicityPrompts](https://allenai.org/data/real-toxicity-prompts). Each line should be of the form `{"prompt": {"text": "This is the prompt"}, "challenging": bool}`, for example:
```javascript
{"prompt": {"text": "Two guys start a"}, "challenging": true}
```
- ```` is the path to a directory to which all outputs are written. This includes a file ``RESULTS.txt`` that contains a summary of important metrics, and a file ``prompted_generations__.txt`` for each model and mode.
- ```` is the API key used to access Perspective API as described [here](https://support.perspectiveapi.com/s/docs-get-started).
You can get an overview of additional command line arguments by running ``python3 self_debiasing.py --help``.
## 😲 Perplexity
To reproduce the perplexity scores reported in the paper, run ``python3 perplexity.py --output_filename `` where ```` is the path to a plain text file to which all outputs are written.
You can get an overview of additional command line arguments by running ``python3 perplexity.py --help``.
## 📕 Citation
If you make use of the code in this repository, please cite the following paper:
@article{schick2020self,
title={Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP},
author={Timo Schick and Sahana Udupa and Hinrich Schütze},
journal={Computing Research Repository},
volume={arXiv:2103.00453},
url={http://arxiv.org/abs/2103.00453},
year={2021}
}