https://github.com/facebookresearch/responsiblenlp

Repository for research in the field of Responsible NLP at Meta.
https://github.com/facebookresearch/responsiblenlp

Last synced: about 21 hours ago
JSON representation

Repository for research in the field of Responsible NLP at Meta.

• Host: GitHub
• URL: https://github.com/facebookresearch/responsiblenlp
• Owner: facebookresearch
• License: other
• Created: 2022-05-17T16:55:15.000Z (over 2 years ago)
• Default Branch: main
• Last Pushed: 2024-08-15T05:31:08.000Z (3 months ago)
• Last Synced: 2024-11-06T12:12:18.966Z (8 days ago)
• Language: Python
• Size: 52.9 MB
• Stars: 183
• Watchers: 21
• Forks: 28
• Open Issues: 6
• Metadata Files:
  • Readme: README.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE
  • Code of conduct: CODE_OF_CONDUCT.md

Awesome Lists containing this project

README

        
This repository hosts code and datasets relating to Responsible NLP projects from Meta AI.

# Projects

- [`AdvPromptSet`](https://github.com/facebookresearch/ResponsibleNLP/tree/main/AdvPromptSet)

  - **AdvPromptSet**: a comprehensive and challenging adversarial text prompt set with 197,628 prompts of varying toxicity levels and more than 24 sensitive demographic identity groups and combinations.

- [`fairscore`](https://github.com/facebookresearch/ResponsibleNLP/tree/main/fairscore):

  - From [Rebecca Qian, Candace Ross, Jude Fernandes, Eric Smith, Douwe Kiela, Adina Williams. *Perturbation Augmentation for Fairer NLP.* 2022.](https://aclanthology.org/2022.emnlp-main.646/)

  - **PANDA**, an annotated dataset of 100K demographic perturbations of diverse text, rewritten to change gender, race/ethnicity and age references.

  - The perturber, pretrained models, code and other artifacts related to the Perturbation Augmentation for Fairer NLP project will be released shortly.

- [`gender_gap_pipeline`](https://github.com/facebookresearch/ResponsibleNLP/tree/main/gender_gap_pipeline):

  - **The Gender-GAP Pipeline**, from [Benjamin Muller,  Belen Alastruey, Prangthip Hansanti, Elahe Kalbassi, Christophe Ropers, Eric Michael Smith, Adina Williams, Luke Zettlemoyer, Pierre Andrews, Marta R Costa-jussà](https://arxiv.org/pdf/2308.16871.pdf)

- [`holistic_bias`](https://github.com/facebookresearch/ResponsibleNLP/tree/main/holistic_bias):

  - From [Eric Michael Smith, Melissa Hall, Melanie Kambadur, Eleonora Presani, Adina Williams. *"I'm sorry to hear that": finding bias in language models with a holistic descriptor dataset.* 2022.](https://arxiv.org/pdf/2205.09209.pdf)

  - Code to generate a dataset, **HolisticBias**, consisting of nearly 600 demographic terms in over 450k sentence prompts

  - Code to calculate **Likelihood Bias**, a metric of the amount of bias in a language model, defined on HolisticBias demographic terms

- [`robbie`](https://github.com/facebookresearch/ResponsibleNLP/tree/main/robbie):

  - **ROBBIE**: we test 6 bias/toxicity metrics (including 2 novel ones) across 5 model families and 3 bias/toxicity mitigation techniques, and show that using a broad array of metrics enables much better assessment of safety issues in these models and mitigations.

-----

See [CONTRIBUTING.md](https://github.com/facebookresearch/ResponsibleNLP/blob/main/CONTRIBUTING.md) for how to help out, and see [LICENSE](https://github.com/facebookresearch/ResponsibleNLP/blob/main/LICENSE) for license information.